Nov 03 17:30:55.950885 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Nov 03 17:30:55.950900 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Nov 03 17:30:55.950909 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'. Nov 03 17:30:55.951359 ip-10-0-139-234 systemd[1]: Failed to start Kubernetes Kubelet. Nov 03 17:31:05.995266 ip-10-0-139-234 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Nov 03 17:31:05.995278 ip-10-0-139-234 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot d820c2d182d14db2a53f58ee86ac66e1 -- Nov 03 17:32:58.403898 ip-10-0-139-234 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:32:58.775976 ip-10-0-139-234 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:32:58.775976 ip-10-0-139-234 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:32:58.775976 ip-10-0-139-234 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:32:58.775976 ip-10-0-139-234 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:32:58.780767 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780679 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.780767 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780763 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.780767 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780768 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.780767 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780774 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780778 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780782 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780785 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780788 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780792 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780796 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780799 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780801 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780804 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780807 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780809 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780812 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780815 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780817 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780820 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780822 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780825 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780827 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780830 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.780946 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780832 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780835 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780838 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780840 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780843 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780846 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780848 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780851 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780854 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780857 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780860 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780862 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780865 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780868 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780871 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780874 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780876 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780880 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780882 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780885 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.781439 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780887 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780890 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780893 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780895 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780898 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780900 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780903 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780907 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780910 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780913 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780915 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780918 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780921 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780923 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780926 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780929 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780932 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780934 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780937 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.781967 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780940 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780942 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780945 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780947 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780950 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780955 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780957 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780961 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780963 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780966 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780969 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780971 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780974 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780976 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780979 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780981 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780984 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780986 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780988 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780991 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.782483 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780993 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.780998 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781000 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781003 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781005 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781008 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781011 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781013 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781016 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781018 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781021 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781024 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781026 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781029 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781031 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781034 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781037 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781040 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781042 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.783020 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781045 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781049 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781052 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781055 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781058 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781852 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781860 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781863 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781865 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781868 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781871 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781873 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781876 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781879 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781882 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781885 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781888 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781891 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781893 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.783530 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781896 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781899 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781903 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781907 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781910 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781913 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781916 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781918 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781921 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781924 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781926 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781929 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781932 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781935 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781938 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781941 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781944 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781946 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781949 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781951 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.784039 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781953 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781956 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781959 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781961 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781964 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781967 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781970 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781972 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781975 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781978 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781981 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781984 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781986 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781989 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781991 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781994 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.781997 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782001 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782004 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782007 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.784563 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782009 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782012 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782015 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782018 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782020 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782023 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782026 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782028 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782031 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782033 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782036 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782039 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782041 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782044 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782046 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782049 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782051 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782054 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782057 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.785091 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782059 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782063 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782068 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782071 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782074 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782077 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782080 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782082 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782085 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782088 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782094 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782097 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782100 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782103 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782105 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782108 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782111 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782114 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782117 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.785629 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782119 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782123 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782126 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782128 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782131 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782134 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782136 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782139 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782141 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782144 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782146 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782149 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782151 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782154 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782673 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782679 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782686 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782689 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782692 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782696 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.786123 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782698 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782702 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782706 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782711 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782715 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782718 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782720 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782723 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782725 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782728 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782731 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782733 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782736 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782738 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782741 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782744 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782746 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782749 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782751 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.786650 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782754 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782756 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782759 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782762 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782764 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782767 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782769 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782772 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782774 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782777 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782780 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782783 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782786 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782789 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782791 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782794 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782798 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782801 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782804 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782807 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.787158 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782809 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782812 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782814 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782817 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782819 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782822 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782824 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782827 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782829 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782832 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782835 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782837 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782841 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782844 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782847 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782849 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782852 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782854 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782857 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782861 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.787671 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782864 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782866 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782868 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782872 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782874 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782877 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782879 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782882 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782885 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782888 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782890 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782893 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782895 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782898 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782900 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782903 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782905 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782908 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782910 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782913 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.788193 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782916 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782918 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782921 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782925 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782930 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782933 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782943 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782947 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782952 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782955 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782959 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782962 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782965 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782967 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782970 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782974 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782976 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782979 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.788729 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782983 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.782987 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784107 2579 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784121 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784128 2579 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784132 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784137 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784140 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784145 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784154 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784157 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784160 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784164 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784167 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784170 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784173 2579 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784176 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784179 2579 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784182 2579 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784185 2579 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784189 2579 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784192 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784195 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784198 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:32:58.789212 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784202 2579 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784206 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784210 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784214 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784217 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784220 2579 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784223 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784226 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784230 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784233 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784238 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784241 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784243 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784246 2579 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784250 2579 flags.go:64] FLAG: --enable-server="true" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784252 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784257 2579 flags.go:64] FLAG: --event-burst="100" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784260 2579 flags.go:64] FLAG: --event-qps="50" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784263 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784266 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784269 2579 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784273 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784276 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784279 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784282 2579 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:32:58.789835 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784285 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784288 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784291 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784294 2579 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784297 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784300 2579 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784303 2579 flags.go:64] FLAG: --feature-gates="" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784307 2579 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784310 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784314 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784317 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784320 2579 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784323 2579 flags.go:64] FLAG: --help="false" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784326 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784330 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784333 2579 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784352 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784356 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784360 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784364 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784367 2579 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784370 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784373 2579 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:32:58.790499 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784376 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784379 2579 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784382 2579 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784385 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784388 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784391 2579 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784394 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784397 2579 flags.go:64] FLAG: --lock-file="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784400 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784403 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784406 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784411 2579 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784418 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784421 2579 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784424 2579 flags.go:64] FLAG: --logging-format="text" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784427 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784431 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784434 2579 flags.go:64] FLAG: --manifest-url="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784437 2579 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784441 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784444 2579 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784449 2579 flags.go:64] FLAG: --max-pods="110" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784452 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784455 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784458 2579 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:32:58.791169 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784461 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784465 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784467 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784470 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784478 2579 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784481 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784484 2579 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784487 2579 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784490 2579 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784493 2579 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784496 2579 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784499 2579 flags.go:64] FLAG: --port="10250" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784502 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784504 2579 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-05a9d469a55b2235a" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784508 2579 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784510 2579 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784513 2579 flags.go:64] FLAG: --register-node="true" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784516 2579 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784520 2579 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784523 2579 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784526 2579 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784528 2579 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784532 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784536 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784539 2579 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784542 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:32:58.791837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784545 2579 flags.go:64] FLAG: --runonce="false" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784547 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784550 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784553 2579 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784556 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784559 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784563 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784581 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784586 2579 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784589 2579 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784592 2579 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784595 2579 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784598 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784601 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784603 2579 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784606 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784612 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784615 2579 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784618 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784621 2579 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784624 2579 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784627 2579 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784630 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784633 2579 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784636 2579 flags.go:64] FLAG: --v="2" Nov 03 17:32:58.792505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784640 2579 flags.go:64] FLAG: --version="false" Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784644 2579 flags.go:64] FLAG: --vmodule="" Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784648 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.784652 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784791 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784796 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784799 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784802 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784805 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784808 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784815 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784818 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784820 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784823 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784827 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784830 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784832 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784835 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.793163 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784840 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784843 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784845 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784848 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784851 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784853 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784856 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784859 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784861 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784864 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784866 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784869 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784871 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784874 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784876 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784879 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784883 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784885 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784888 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.793726 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784891 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784894 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784896 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784899 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784902 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784905 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784907 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784910 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784914 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784917 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784920 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784922 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784925 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784927 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784930 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784932 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784935 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784937 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784940 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784943 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.794225 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784945 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784949 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784953 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784956 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784959 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784962 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784964 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784967 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784970 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784974 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784976 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784979 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784983 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784985 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784988 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784990 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784994 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.784998 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785000 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785003 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.794797 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785005 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785008 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785011 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785014 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785016 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785019 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785021 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785024 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785026 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785029 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785031 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785034 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785037 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785039 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785042 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785045 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785047 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785049 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785052 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.795357 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785055 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785057 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785061 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785064 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785066 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785069 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785071 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785074 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785076 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785080 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785084 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785088 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.785114 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785150 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785155 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.795855 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785159 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785163 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785166 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785168 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785171 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785174 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785176 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785179 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785182 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785184 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785187 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785190 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785192 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785195 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785197 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785200 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785203 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785205 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785208 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785211 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.796235 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785214 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785217 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785219 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785222 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785224 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785227 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785229 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785233 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785235 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785238 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785240 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785243 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785245 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785249 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785252 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785254 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785257 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785259 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785262 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.796761 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785264 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785267 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785269 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785272 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785274 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785277 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785279 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785282 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785284 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785289 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785293 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785296 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785300 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785303 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785306 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785309 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785312 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785314 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785317 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.797246 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785319 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785323 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785326 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785328 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785331 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785334 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785336 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785339 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785342 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785345 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785347 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785350 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785352 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785355 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785358 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785360 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785363 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785365 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785368 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.797935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785370 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785373 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785376 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785378 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785381 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785383 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785387 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785390 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785392 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785395 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785397 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785400 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785402 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785405 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785409 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785411 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785414 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785416 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785419 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.798485 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785422 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785424 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785426 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785429 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785433 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785436 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785438 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.785441 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.792831 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.792847 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.793550 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.793557 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793654 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793661 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793665 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793668 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793671 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793674 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793677 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.798999 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793680 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793683 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793685 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793688 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793690 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793693 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793696 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793698 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793701 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793703 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793706 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793708 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793711 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793714 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793717 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793719 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793728 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793731 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793735 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793737 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.799487 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793740 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793743 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793746 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793749 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793751 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793754 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793757 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793759 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793762 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793765 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793767 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793770 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793772 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793777 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793780 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793783 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793786 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793789 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793791 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.800010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793794 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793797 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793799 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793802 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793804 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793807 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793809 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793812 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793814 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793817 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793828 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793831 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793834 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793837 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793840 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793843 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793846 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793848 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793851 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793854 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.800496 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793856 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793859 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793861 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793864 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793866 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793869 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793871 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793874 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793876 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793879 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793882 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793884 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793887 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793890 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793893 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793896 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793898 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793901 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793903 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793906 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.801053 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793908 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793911 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793914 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793923 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793926 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793929 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793932 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793934 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793937 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793939 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793942 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793944 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793947 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793949 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793952 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793954 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793957 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793960 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793964 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.801627 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.793968 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.793996 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794197 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794202 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794205 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794209 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794211 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794214 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794216 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794219 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794221 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794224 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794227 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794230 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794234 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794238 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.802115 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794242 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794251 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794254 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794257 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794259 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794262 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794264 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794267 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794269 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794272 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794274 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794277 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794280 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794282 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794285 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794287 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794290 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794292 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794295 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794297 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.802545 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794300 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794302 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794305 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794307 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794310 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794312 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794315 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794318 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794320 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794323 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794325 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794328 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794331 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794333 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794342 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794345 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794347 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794350 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794352 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.803290 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794354 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794357 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794359 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794362 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794364 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794367 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794369 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794372 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794375 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794377 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794380 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794382 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794386 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794390 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794393 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794395 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794398 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794400 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.804133 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794405 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794408 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794410 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794413 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794415 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794418 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794420 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794423 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794425 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794433 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794436 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794439 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794441 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794444 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794446 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794449 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794452 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794454 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794457 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794460 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.804706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794463 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794465 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794468 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794470 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794473 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794476 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794478 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794481 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794483 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794485 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794488 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794491 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794493 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794495 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.794518 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794561 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:32:58.805405 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794565 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794582 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794586 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794589 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794592 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794595 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794610 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794613 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794616 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794618 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794621 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794623 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794626 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794628 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794631 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794633 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794636 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794638 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794641 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:32:58.805929 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794644 2579 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794646 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794649 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794651 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794654 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794656 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794659 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794661 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794664 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794667 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794669 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794671 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794674 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794676 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794679 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794681 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794684 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794687 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794690 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:32:58.806415 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794692 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794700 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794703 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794706 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794709 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794711 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794714 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794716 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794719 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794722 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794724 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794726 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794729 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794731 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794734 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794736 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794740 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794744 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794747 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:32:58.807010 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794749 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794752 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794754 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794757 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794760 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794762 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794765 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794767 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794771 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794773 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794776 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794778 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794781 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794786 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794788 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794796 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794799 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794802 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794805 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:32:58.807494 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794807 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794810 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794813 2579 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794815 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794818 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794820 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794823 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794825 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794828 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794830 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794833 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794836 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794839 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794841 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794844 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794846 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794849 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794851 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794853 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794856 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:32:58.807998 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794863 2579 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794866 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794868 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794871 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794873 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794876 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794880 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794882 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:32:58.794885 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.794949 2579 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.797313 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.798172 2579 server.go:1014] "Starting client certificate rotation" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.798285 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.798323 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.801525 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:32:58.808505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.805782 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:32:58.818837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.818817 2579 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:32:58.822317 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.822300 2579 log.go:25] "Validated CRI v1 image API" Nov 03 17:32:58.824012 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.823985 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:32:58.827011 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.826988 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b02b216c-d8cf-4dd6-b4fc-9a69cab1708e:/dev/nvme0n1p4 d2f9b9a6-2918-4012-8830-47cd2ad8ac7f:/dev/nvme0n1p3] Nov 03 17:32:58.827085 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.827011 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Nov 03 17:32:58.832451 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.832335 2579 manager.go:217] Machine: {Timestamp:2026-11-03 17:32:58.830975486 +0000 UTC m=+0.320477655 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100016 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e167626d6e2779c16d1c346cc9bd SystemUUID:ec26e167-626d-6e27-79c1-6d1c346cc9bd BootID:d820c2d1-82d1-4db2-a53f-58ee86ac66e1 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0f:d8:52:7a:4b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0f:d8:52:7a:4b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:5e:c6:6a:1f:ec:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:32:58.832538 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.832453 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:32:58.832538 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.832526 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:32:58.835101 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835079 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:32:58.835243 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835104 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-234.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:32:58.835286 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835246 2579 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:32:58.835286 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835254 2579 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:32:58.835342 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835285 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:32:58.835342 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.835305 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:32:58.836489 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.836479 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:32:58.840830 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.840817 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:32:58.840952 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.840943 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:32:58.841725 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.841715 2579 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:32:58.841756 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.841730 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:32:58.841756 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.841740 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:32:58.841756 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.841749 2579 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:32:58.841756 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.841757 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:32:58.842092 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.842079 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:32:58.845068 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.845052 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:32:58.852100 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.852088 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:32:58.852972 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.852961 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:32:58.853008 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.852990 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:32:58.853174 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853161 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:32:58.853239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853181 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:32:58.853239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853192 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:32:58.853239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853201 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:32:58.853239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853215 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:32:58.853239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853224 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:32:58.853889 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853879 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:32:58.853921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853891 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:32:58.853921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853899 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:32:58.853921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853905 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:32:58.853921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853913 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:32:58.853921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.853922 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:32:58.854667 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.854658 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:32:58.854667 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.854668 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:32:58.858715 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.858692 2579 server.go:1263] "Started kubelet" Nov 03 17:32:58.858802 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.858757 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:32:58.858850 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.858788 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:32:58.858897 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.858876 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:32:58.859497 ip-10-0-139-234 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:32:58.862779 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.862758 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:32:58.865652 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.865632 2579 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:32:58.867764 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:58.867748 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Nov 03 17:32:58.871322 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.871303 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:32:58.871884 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.871864 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:32:58.872058 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:58.872040 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:32:58.872800 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872784 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:32:58.872800 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872787 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:32:58.872916 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872824 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:32:58.872916 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872915 2579 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:32:58.872969 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872918 2579 factory.go:55] Registering systemd factory Nov 03 17:32:58.872969 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872933 2579 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:32:58.873031 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.872920 2579 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:32:58.873139 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.873127 2579 factory.go:153] Registering CRI-O factory Nov 03 17:32:58.873198 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.873141 2579 factory.go:223] Registration of the crio container factory successfully Nov 03 17:32:58.873198 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.873195 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:32:58.873251 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.873213 2579 factory.go:103] Registering Raw factory Nov 03 17:32:58.873251 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.873229 2579 manager.go:1199] Started watching for new ooms in manager Nov 03 17:32:58.874367 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.874338 2579 manager.go:319] Starting recovery of all containers Nov 03 17:32:58.885727 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.885713 2579 manager.go:324] Recovery completed Nov 03 17:32:58.889537 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.889525 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:58.895210 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895186 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:58.895286 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895223 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:58.895286 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895236 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:58.895800 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895783 2579 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:32:58.895800 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895797 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:32:58.895968 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895824 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:58.896157 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.895817 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:32:58.897152 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.897128 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:58.897242 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.897171 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:58.897242 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.897187 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:58.898545 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.898527 2579 policy_none.go:50] "Start" Nov 03 17:32:58.898652 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.898562 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:32:58.898652 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.898597 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:32:58.900683 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.900670 2579 policy_none.go:44] "Start" Nov 03 17:32:58.939138 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.939109 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:32:58.941625 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.941614 2579 manager.go:341] "Starting Device Plugin manager" Nov 03 17:32:58.941666 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:58.941647 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:32:58.941666 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.941658 2579 server.go:87] "Starting device plugin registration server" Nov 03 17:32:58.941936 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.941921 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:32:58.942000 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.941938 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:32:58.942045 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.942005 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:32:58.942107 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.942094 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:32:58.942107 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:58.942106 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:32:58.942520 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:58.942496 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:32:58.942625 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:58.942561 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:32:59.007184 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.007026 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-234.us-west-1.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Nov 03 17:32:59.007184 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.007028 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-234.us-west-1.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Nov 03 17:32:59.019227 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.019205 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:32:59.020550 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.020530 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:32:59.020550 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.020549 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:32:59.020728 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.020582 2579 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:32:59.020728 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.020624 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Nov 03 17:32:59.042482 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.042434 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:59.043236 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.043218 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:59.043297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.043251 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:59.043297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.043264 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:59.043297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.043288 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.056868 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.056125 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-234.us-west-1.compute.internal.18e4995d97d2ebc1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:32:58.858490817 +0000 UTC m=+0.347992986,LastTimestamp:2026-11-03 17:32:58.858490817 +0000 UTC m=+0.347992986,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.us-west-1.compute.internal,}" Nov 03 17:32:59.089614 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.089590 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:32:59.089690 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.089650 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-s7n4w" Nov 03 17:32:59.120702 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.120682 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal"] Nov 03 17:32:59.120782 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.120748 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:59.121652 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.121638 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:59.121724 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.121664 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:59.121724 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.121676 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:59.123709 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.123698 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:59.123870 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.123850 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.123921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.123885 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:59.124655 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124639 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:59.124725 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124663 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:59.124725 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124681 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:59.124803 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124679 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:59.124803 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124749 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:59.124803 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.124770 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:59.126697 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.126683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.126772 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.126709 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:32:59.127488 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.127474 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:32:59.127564 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.127513 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:32:59.127564 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.127527 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:32:59.144423 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.144402 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.146662 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.146646 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.173628 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.173611 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:32:59.174522 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.174499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.174639 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.174527 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.174639 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.174546 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.271480 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.271454 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:32:59.271987 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.271971 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:32:59.272471 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.272457 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:32:59.273616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.273603 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.273679 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.273623 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-234.us-west-1.compute.internal\": node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:32:59.274877 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274862 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.274924 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274886 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.274924 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274903 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.275005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274969 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.275005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274970 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.275005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.274983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.372664 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.372603 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.446656 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.446632 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.449500 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.449486 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.497148 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.497126 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:32:59.497753 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.497741 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:32:59.557268 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.557243 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:32:59.797887 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.797861 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Nov 03 17:32:59.798443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.798038 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 17:32:59.798443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.798062 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Nov 03 17:32:59.798443 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:32:59.798044 2579 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://aa49546cbf5c9454088617b8e9f7adf8-66509c371b41a33e.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-139-234.us-west-1.compute.internal.18e4995d9a03aad9\": read tcp 10.0.139.234:42118->3.149.82.99:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-139-234.us-west-1.compute.internal.18e4995d9a03aad9 default 4752 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-234.us-west-1.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-139-234.us-west-1.compute.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-139-234.us-west-1.compute.internal,},FirstTimestamp:2026-11-03 17:32:58 +0000 UTC,LastTimestamp:2026-11-03 17:32:59.043267731 +0000 UTC m=+0.532769899,Count:3,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-234.us-west-1.compute.internal,}" Nov 03 17:32:59.842743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.842723 2579 apiserver.go:52] "Watching apiserver" Nov 03 17:32:59.872185 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:32:59.872157 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Nov 03 17:33:00.093912 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.093824 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:27:59 +0000 UTC" deadline="2028-07-23 15:53:59.954325666 +0000 UTC" Nov 03 17:33:00.093912 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.093878 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15070h20m59.860451743s" Nov 03 17:33:00.106839 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.106815 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:33:00.107135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.107118 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-8jlfb","openshift-ovn-kubernetes/ovnkube-node-d8hsc","openshift-multus/multus-additional-cni-plugins-xmdk5","openshift-network-diagnostics/network-check-target-xp42v","openshift-network-operator/iptables-alerter-hp94c","kube-system/konnectivity-agent-vwkww","openshift-image-registry/node-ca-gv7lt","openshift-multus/network-metrics-daemon-tlq8b","kube-system/global-pull-secret-syncer-fnq58","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm","openshift-dns/node-resolver-gkxfp","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal","openshift-multus/multus-knk2x","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal"] Nov 03 17:33:00.109341 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.109317 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.111275 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.111258 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.113209 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.113196 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.115092 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.115077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:00.115155 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.115141 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:00.117098 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.117079 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.119068 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.119053 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.119154 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.119115 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:00.121020 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.121004 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.123196 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.123177 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.125314 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.125295 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.127488 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.127465 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.129721 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.129703 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.129808 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.129768 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:00.131909 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.131894 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.164170 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164148 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.164388 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164370 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:33:00.164388 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164377 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.164523 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164393 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.164523 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164416 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\"" Nov 03 17:33:00.164523 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164500 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\"" Nov 03 17:33:00.164676 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.164539 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.166237 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166217 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\"" Nov 03 17:33:00.166413 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166398 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:33:00.166466 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166406 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.166466 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166418 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:33:00.166466 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166410 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:33:00.166466 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.166445 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.170014 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.169999 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:33:00.170189 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.170177 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:33:00.173366 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.173347 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\"" Nov 03 17:33:00.173803 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.173788 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:33:00.178804 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178779 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.178873 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178812 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.178873 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178828 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.178873 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178842 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.178873 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178864 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.179010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178895 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.179010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178929 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.179010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178971 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.178992 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179076 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gr8jh\" (UniqueName: \"kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179129 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179157 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179197 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179216 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179271 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179286 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:00.179332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179315 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179338 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179360 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179420 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179439 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179454 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179468 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179482 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179501 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179516 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179530 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179549 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.179563 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179589 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179604 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnhsz\" (UniqueName: \"kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179654 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q496r\" (UniqueName: \"kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179677 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179697 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179718 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179738 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179751 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179764 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179777 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179791 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txrr7\" (UniqueName: \"kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179806 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179829 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179878 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-695fn\" (UniqueName: \"kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.180135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sbl5\" (UniqueName: \"kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.179990 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180051 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180070 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180092 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180115 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180130 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8z6h\" (UniqueName: \"kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180175 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180193 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180246 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180279 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180304 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.180879 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180327 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180372 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgzgp\" (UniqueName: \"kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2hnz\" (UniqueName: \"kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180423 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180444 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180522 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180545 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180612 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180636 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.181330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.180659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.267117 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267072 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Nov 03 17:33:00.267244 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267144 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:33:00.267244 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267157 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:33:00.267244 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267145 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-tf84w" Nov 03 17:33:00.267244 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267240 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267247 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267245 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267313 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267359 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267361 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267389 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267404 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:33:00.267443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267426 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.267745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267496 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:33:00.267745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267507 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:00.267745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267560 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:00.267745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.267617 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\"" Nov 03 17:33:00.268070 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.268059 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\"" Nov 03 17:33:00.270759 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.270745 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\"" Nov 03 17:33:00.281062 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281043 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.281124 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281065 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.281124 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281082 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.281124 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281096 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.281124 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281110 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281124 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281146 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281166 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281188 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281189 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281226 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281244 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281245 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281259 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281269 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281280 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.281297 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281258 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281332 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281312 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281354 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281381 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281405 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281429 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281448 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281440 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281489 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281532 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281545 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281551 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.281921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281589 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281613 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281640 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhsz\" (UniqueName: \"kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281614 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q496r\" (UniqueName: \"kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281695 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281711 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281723 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281737 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281789 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281815 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281795 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281841 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281864 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281888 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281903 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.282619 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281937 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrr7\" (UniqueName: \"kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.281989 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282012 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695fn\" (UniqueName: \"kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282037 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282054 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282146 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282162 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282244 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282689 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282777 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\"" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.282836 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\"" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283039 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283118 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.283464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283446 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbl5\" (UniqueName: \"kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283534 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283594 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283613 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283671 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283734 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.283785 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283794 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283664 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.283883 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:00.783843903 +0000 UTC m=+2.273346081 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8z6h\" (UniqueName: \"kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.283991 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284024 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284056 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.284261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284088 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284178 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284212 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284242 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284273 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzgp\" (UniqueName: \"kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284306 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hnz\" (UniqueName: \"kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284334 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284368 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284441 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284461 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284468 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284477 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284643 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285053 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284681 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284746 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284784 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284816 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284907 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284938 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284984 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285058 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285166 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285176 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285234 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285618 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285788 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.285895 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285847 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285936 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.285955 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.284996 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286004 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286025 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286041 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286068 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286105 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286147 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.286242 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.286308 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:00.786288122 +0000 UTC m=+2.275790279 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286070 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286106 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286464 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.286661 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286136 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286623 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286646 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286665 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286691 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286724 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8jh\" (UniqueName: \"kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286755 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286775 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286781 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286825 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286876 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.287378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.286997 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.287984 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.287398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.287984 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.287405 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.287984 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.287457 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.287984 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.287542 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.289084 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.289065 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.289298 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.289272 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.343903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.343874 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:00.343903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.343895 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:00.343903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.343905 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:00.344085 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.343958 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:00.843944226 +0000 UTC m=+2.333446395 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:00.346430 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.346407 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrr7\" (UniqueName: \"kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.349439 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.349398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbl5\" (UniqueName: \"kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.349773 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.349748 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q496r\" (UniqueName: \"kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.349773 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.349761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhsz\" (UniqueName: \"kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.352044 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.352021 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-695fn\" (UniqueName: \"kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.352937 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.352910 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hnz\" (UniqueName: \"kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.353168 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.353152 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8z6h\" (UniqueName: \"kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.358585 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.358553 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8jh\" (UniqueName: \"kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.359304 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.359280 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzgp\" (UniqueName: \"kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.440603 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.440559 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:00.452716 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.452696 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:00.472560 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.472539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:00.476120 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.476102 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:00.487952 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.487929 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-knk2x" Nov 03 17:33:00.522378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.522358 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:00.524522 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.524502 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:00.526810 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.526794 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:00.528896 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.528882 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:00.702927 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:00.702877 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b39301f54ab728a5f22c5154db8ce29.slice/crio-575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407 WatchSource:0}: Error finding container 575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407: Status 404 returned error can't find the container with id 575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407 Nov 03 17:33:00.710231 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.710211 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:33:00.714109 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:00.714086 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d0bc0f39e3274106efc0e449e45f7e.slice/crio-72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06 WatchSource:0}: Error finding container 72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06: Status 404 returned error can't find the container with id 72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06 Nov 03 17:33:00.791236 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.791206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:00.791371 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.791250 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:00.791411 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.791367 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:00.791411 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.791367 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:00.791479 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.791432 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:01.791413556 +0000 UTC m=+3.280915728 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:00.791479 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.791448 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:01.79144103 +0000 UTC m=+3.280943202 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:00.841984 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.841958 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:00.891625 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:00.891521 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:00.891774 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.891690 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:00.891774 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.891711 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:00.891774 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.891723 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:00.891872 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:00.891777 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:01.891763897 +0000 UTC m=+3.381266055 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:01.025974 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.025911 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"32d0bc0f39e3274106efc0e449e45f7e","Type":"ContainerStarted","Data":"72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06"} Nov 03 17:33:01.026951 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.026918 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerStarted","Data":"575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407"} Nov 03 17:33:01.270360 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.270318 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:00 +0000 UTC" deadline="2028-06-23 02:25:44.133515488 +0000 UTC" Nov 03 17:33:01.270360 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.270348 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14336h52m42.863170116s" Nov 03 17:33:01.314985 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.314950 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62660df0_cd00_4e15_87ce_7d8177463419.slice/crio-788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c WatchSource:0}: Error finding container 788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c: Status 404 returned error can't find the container with id 788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c Nov 03 17:33:01.360950 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.360927 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:01.375118 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.375076 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad57e2f_34f9_4ad6_82d7_3ad939d0fef4.slice/crio-8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0 WatchSource:0}: Error finding container 8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0: Status 404 returned error can't find the container with id 8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0 Nov 03 17:33:01.375921 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.375880 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92dc67cc_a063_43af_9a22_432e61ee10f4.slice/crio-ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd WatchSource:0}: Error finding container ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd: Status 404 returned error can't find the container with id ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd Nov 03 17:33:01.390905 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.390881 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1796bb1_d79a_448a_9152_8570b92e43e6.slice/crio-f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e WatchSource:0}: Error finding container f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e: Status 404 returned error can't find the container with id f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e Nov 03 17:33:01.415819 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.415796 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645841ea_aa0f_4baa_8923_a460dc301b5a.slice/crio-94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc WatchSource:0}: Error finding container 94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc: Status 404 returned error can't find the container with id 94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc Nov 03 17:33:01.428304 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.428282 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32a05a82_dc96_44b1_b348_bef87dcc963c.slice/crio-29cc6e36701b25a019bd337992260d632303387d2b745a5ca75217c1b5e864b3 WatchSource:0}: Error finding container 29cc6e36701b25a019bd337992260d632303387d2b745a5ca75217c1b5e864b3: Status 404 returned error can't find the container with id 29cc6e36701b25a019bd337992260d632303387d2b745a5ca75217c1b5e864b3 Nov 03 17:33:01.442377 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.442354 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe747a0c_5a1d_4db3_876a_f26e1852450e.slice/crio-55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da WatchSource:0}: Error finding container 55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da: Status 404 returned error can't find the container with id 55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da Nov 03 17:33:01.447621 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.447602 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f4f29a5_0dfb_4515_9bd6_276c084934bb.slice/crio-7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718 WatchSource:0}: Error finding container 7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718: Status 404 returned error can't find the container with id 7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718 Nov 03 17:33:01.451085 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:01.451065 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7f26304_7fb0_49b7_8f59_d8dc02c88d09.slice/crio-c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705 WatchSource:0}: Error finding container c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705: Status 404 returned error can't find the container with id c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705 Nov 03 17:33:01.799608 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.799559 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:01.799820 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.799656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:01.799820 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.799806 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:01.799962 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.799865 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.799848426 +0000 UTC m=+5.289350583 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:01.799962 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.799917 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:01.799962 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.799944 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.799931909 +0000 UTC m=+5.289434081 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:01.900748 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:01.900709 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:01.901371 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.900906 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:01.901371 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.900933 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:01.901371 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.900947 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:01.901371 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:01.901017 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:03.900997228 +0000 UTC m=+5.390499401 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:02.021330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.021289 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:02.021330 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.021333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:02.021608 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.021333 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:02.021608 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:02.021443 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:02.021608 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:02.021522 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:02.021730 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:02.021603 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:02.030026 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.029993 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwkww" event={"ID":"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4","Type":"ContainerStarted","Data":"8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0"} Nov 03 17:33:02.031474 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.031441 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gv7lt" event={"ID":"62660df0-cd00-4e15-87ce-7d8177463419","Type":"ContainerStarted","Data":"788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c"} Nov 03 17:33:02.032987 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.032962 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knk2x" event={"ID":"a7f26304-7fb0-49b7-8f59-d8dc02c88d09","Type":"ContainerStarted","Data":"c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705"} Nov 03 17:33:02.034515 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.034490 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hp94c" event={"ID":"8f4f29a5-0dfb-4515-9bd6-276c084934bb","Type":"ContainerStarted","Data":"7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718"} Nov 03 17:33:02.035989 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.035956 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" event={"ID":"d1796bb1-d79a-448a-9152-8570b92e43e6","Type":"ContainerStarted","Data":"f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e"} Nov 03 17:33:02.037662 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.037641 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da"} Nov 03 17:33:02.039051 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.039026 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" event={"ID":"32a05a82-dc96-44b1-b348-bef87dcc963c","Type":"ContainerStarted","Data":"29cc6e36701b25a019bd337992260d632303387d2b745a5ca75217c1b5e864b3"} Nov 03 17:33:02.040353 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.040325 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkxfp" event={"ID":"645841ea-aa0f-4baa-8923-a460dc301b5a","Type":"ContainerStarted","Data":"94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc"} Nov 03 17:33:02.041605 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.041563 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerStarted","Data":"ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd"} Nov 03 17:33:02.271127 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.271019 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:00 +0000 UTC" deadline="2028-08-18 21:06:59.13938438 +0000 UTC" Nov 03 17:33:02.271127 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:02.271119 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15699h33m56.868270025s" Nov 03 17:33:03.818332 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:03.818287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:03.819052 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:03.818400 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:03.819052 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.818533 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:03.819052 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.818597 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:07.81857799 +0000 UTC m=+9.308080164 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:03.819052 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.818687 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:03.819052 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.818729 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:07.818716541 +0000 UTC m=+9.308218719 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:03.919583 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:03.919543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:03.919787 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.919714 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:03.919787 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.919733 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:03.919787 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.919744 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:03.920115 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:03.919792 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:07.919778963 +0000 UTC m=+9.409281136 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:04.021261 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:04.021223 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:04.021466 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:04.021353 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:04.021466 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:04.021422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:04.021611 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:04.021477 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:04.021611 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:04.021506 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:04.021611 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:04.021551 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.021392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.021649 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:06.021814 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.021858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:06.021935 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:06.022017 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:06.022007 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:06.071423 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.071380 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="5218a1651407dd57bbe0feff2cf21b6b43f9b718c4881383fba486832d2f5128" exitCode=0 Nov 03 17:33:06.071589 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.071490 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"5218a1651407dd57bbe0feff2cf21b6b43f9b718c4881383fba486832d2f5128"} Nov 03 17:33:06.076028 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.076000 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"32d0bc0f39e3274106efc0e449e45f7e","Type":"ContainerStarted","Data":"1055ebab309ff89dd078386a315b93a1d201a84963fa53c40a1d005c54021bcd"} Nov 03 17:33:06.079567 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.079539 2579 generic.go:356] "Generic (PLEG): container finished" podID="4b39301f54ab728a5f22c5154db8ce29" containerID="931e553d221910359c664069ca9590512091222e8bf62c858db9e7f98d775e5f" exitCode=0 Nov 03 17:33:06.079697 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.079584 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerDied","Data":"931e553d221910359c664069ca9590512091222e8bf62c858db9e7f98d775e5f"} Nov 03 17:33:06.419864 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:06.418825 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" podStartSLOduration=7.418807756 podStartE2EDuration="7.418807756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:06.418424633 +0000 UTC m=+7.907926812" watchObservedRunningTime="2026-11-03 17:33:06.418807756 +0000 UTC m=+7.908309936" Nov 03 17:33:07.088315 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.088272 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-vwkww" event={"ID":"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4","Type":"ContainerStarted","Data":"e815b960de43994ba5c8b5cc5f1e6aea376f976653c39bee7723dc4f3840d3a5"} Nov 03 17:33:07.095345 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.095320 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerStarted","Data":"acb7bf89559a3afdadd8307716d2a8477e17d61b227d0710b7c332ac2c266052"} Nov 03 17:33:07.326667 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.326526 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" podStartSLOduration=8.326506978 podStartE2EDuration="8.326506978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:07.326310362 +0000 UTC m=+8.815812534" watchObservedRunningTime="2026-11-03 17:33:07.326506978 +0000 UTC m=+8.816009157" Nov 03 17:33:07.328089 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.327272 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-vwkww" podStartSLOduration=3.164545305 podStartE2EDuration="8.32726104s" totalImagesPullingTime="5.162715735s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:07.208628984 +0000 UTC m=+8.698131163" watchObservedRunningTime="2026-11-03 17:33:07.32726104 +0000 UTC m=+8.816763219" Nov 03 17:33:07.857474 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.857439 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:07.857662 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.857497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:07.857662 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.857632 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:07.857791 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.857690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:15.857671193 +0000 UTC m=+17.347173366 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:07.857791 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.857752 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:07.857791 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.857786 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:15.857774185 +0000 UTC m=+17.347276355 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:07.958240 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:07.958205 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:07.958453 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.958434 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:07.958512 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.958462 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:07.958512 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.958476 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:07.958611 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:07.958585 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:15.958512891 +0000 UTC m=+17.448015062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:08.021084 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:08.021049 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:08.021258 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:08.021198 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:08.021321 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:08.021269 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:08.021369 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:08.021338 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:08.021421 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:08.021386 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:08.021476 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:08.021451 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:08.103590 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:08.103538 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-gv7lt" event={"ID":"62660df0-cd00-4e15-87ce-7d8177463419","Type":"ContainerStarted","Data":"383296f19139ec8bf81368e50e6d95042489dc65fa8332fa8fb9accf8a200b13"} Nov 03 17:33:09.109073 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:09.109041 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" event={"ID":"d1796bb1-d79a-448a-9152-8570b92e43e6","Type":"ContainerStarted","Data":"c80cc835cac6ec8906cd5685e704b5fc8021b8c71f4aa6ea7fc85e005d3b285f"} Nov 03 17:33:10.021735 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.021706 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:10.021907 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:10.021840 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:10.022160 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.022145 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:10.022234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:10.022219 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:10.022270 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.022256 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:10.022327 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:10.022315 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:10.663324 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.663270 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:33:10.909750 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.909721 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:10.910367 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.910345 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:10.951907 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.951793 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:33:10.663292094Z","UUID":"d681db47-968c-4721-9986-9d6af83c0302","Handler":null,"Name":"","Endpoint":""} Nov 03 17:33:10.956171 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.956138 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:33:10.956305 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:10.956182 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:33:11.027938 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.027444 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-gv7lt" podStartSLOduration=5.999113322 podStartE2EDuration="12.027426687s" totalImagesPullingTime="6.028313365s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:08.221097437 +0000 UTC m=+9.710599618" watchObservedRunningTime="2026-11-03 17:33:11.027426687 +0000 UTC m=+12.516928864" Nov 03 17:33:11.114944 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.114906 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="b562b8dec51e6b5d85b4156af6822cf37fff453cd8e04e8dd4bb0de03a310848" exitCode=0 Nov 03 17:33:11.115092 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.115001 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"b562b8dec51e6b5d85b4156af6822cf37fff453cd8e04e8dd4bb0de03a310848"} Nov 03 17:33:11.117512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.117483 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-knk2x" event={"ID":"a7f26304-7fb0-49b7-8f59-d8dc02c88d09","Type":"ContainerStarted","Data":"4fe6d7c938609f9e623902ba749bdc2625dacd74d132202819192116ea55957f"} Nov 03 17:33:11.119731 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.119707 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" event={"ID":"d1796bb1-d79a-448a-9152-8570b92e43e6","Type":"ContainerStarted","Data":"2452372bf88f9c5ea000af9396dc331a969ded813467b6c63e4d6c21e65f2241"} Nov 03 17:33:11.121645 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.121620 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gkxfp" event={"ID":"645841ea-aa0f-4baa-8923-a460dc301b5a","Type":"ContainerStarted","Data":"965269a8c1e69b5db0bc37d89ceb1a7950584a394bc9a185c2abce0c0ad722b8"} Nov 03 17:33:11.121774 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.121753 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:11.122217 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.122199 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:11.377800 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.377697 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-gkxfp" podStartSLOduration=3.591925092 podStartE2EDuration="12.377676948s" totalImagesPullingTime="8.785751856s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:11.376738398 +0000 UTC m=+12.866240575" watchObservedRunningTime="2026-11-03 17:33:11.377676948 +0000 UTC m=+12.867179127" Nov 03 17:33:11.615883 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:11.615797 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-knk2x" podStartSLOduration=4.007678338 podStartE2EDuration="12.61578195s" totalImagesPullingTime="8.608103612s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:11.615690784 +0000 UTC m=+13.105192963" watchObservedRunningTime="2026-11-03 17:33:11.61578195 +0000 UTC m=+13.105284128" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:12.022406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:12.022518 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:12.022613 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:12.022689 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:12.022746 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:12.023112 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:12.022814 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:12.126027 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:12.125986 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-hp94c" event={"ID":"8f4f29a5-0dfb-4515-9bd6-276c084934bb","Type":"ContainerStarted","Data":"69388b8dc76d5316ac7b7c0e660de3942c3e1843f569b6e5c62c355ed4e8aadd"} Nov 03 17:33:12.247186 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:12.247128 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-hp94c" podStartSLOduration=3.178345037 podStartE2EDuration="12.247109574s" totalImagesPullingTime="9.068764537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:33:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:12.24616046 +0000 UTC m=+13.735662639" watchObservedRunningTime="2026-11-03 17:33:12.247109574 +0000 UTC m=+13.736611752" Nov 03 17:33:13.130607 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.130348 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" event={"ID":"d1796bb1-d79a-448a-9152-8570b92e43e6","Type":"ContainerStarted","Data":"29e0e03e1fc232f39b0730cc736f10f13c41326ba8f4ebe4f3ffd49499447d75"} Nov 03 17:33:13.132500 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.132475 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" event={"ID":"32a05a82-dc96-44b1-b348-bef87dcc963c","Type":"ContainerStarted","Data":"ad3dca3307e18945583847d4298d7e608770e8c9f391b90a240db1b192bd641b"} Nov 03 17:33:13.135223 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.134822 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="3a1ed2b9fe770ae56f05d3cee5d5948170623b8c07a99808eb2303e25f9ea21e" exitCode=0 Nov 03 17:33:13.135223 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.134910 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"3a1ed2b9fe770ae56f05d3cee5d5948170623b8c07a99808eb2303e25f9ea21e"} Nov 03 17:33:13.253180 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.253128 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" podStartSLOduration=3.8392336240000002 podStartE2EDuration="14.253113734s" totalImagesPullingTime="10.41388011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:13.253050155 +0000 UTC m=+14.742552337" watchObservedRunningTime="2026-11-03 17:33:13.253113734 +0000 UTC m=+14.742615926" Nov 03 17:33:13.400396 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:13.400296 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" podStartSLOduration=3.369064454 podStartE2EDuration="14.400278825s" totalImagesPullingTime="11.031214371s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:13.399303578 +0000 UTC m=+14.888805756" watchObservedRunningTime="2026-11-03 17:33:13.400278825 +0000 UTC m=+14.889781000" Nov 03 17:33:14.020966 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:14.020932 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:14.021140 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:14.020930 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:14.021140 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:14.021070 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:14.021140 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:14.020933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:14.021140 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:14.021118 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:14.021357 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:14.021201 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:15.142561 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:15.142527 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="cd2832c39fae35844832a60f4199e45d792ad35aa717ffaf03736a4d27b33c4a" exitCode=0 Nov 03 17:33:15.143302 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:15.142599 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"cd2832c39fae35844832a60f4199e45d792ad35aa717ffaf03736a4d27b33c4a"} Nov 03 17:33:15.919065 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:15.919032 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:15.919256 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:15.919074 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:15.919256 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:15.919170 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:15.919256 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:15.919172 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:15.919256 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:15.919223 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret podName:fedfb5ca-94b5-4850-8020-a7279aa49eae nodeName:}" failed. No retries permitted until 2026-11-03 17:33:31.919206862 +0000 UTC m=+33.408709034 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret") pod "global-pull-secret-syncer-fnq58" (UID: "fedfb5ca-94b5-4850-8020-a7279aa49eae") : object "kube-system"/"original-pull-secret" not registered Nov 03 17:33:15.919256 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:15.919242 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:31.919232745 +0000 UTC m=+33.408734904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : object "openshift-multus"/"metrics-daemon-secret" not registered Nov 03 17:33:16.020234 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:16.020201 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:16.020369 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.020333 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Nov 03 17:33:16.020369 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.020347 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Nov 03 17:33:16.020369 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.020356 2579 projected.go:196] Error preparing data for projected volume kube-api-access-9b22p for pod openshift-network-diagnostics/network-check-target-xp42v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:16.020495 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.020399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p podName:333b0df1-e4ba-4064-979c-655f7e52b9f0 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.020386217 +0000 UTC m=+33.509888386 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-9b22p" (UniqueName: "kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p") pod "network-check-target-xp42v" (UID: "333b0df1-e4ba-4064-979c-655f7e52b9f0") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Nov 03 17:33:16.021237 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:16.021211 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:16.021391 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:16.021303 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:16.021459 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:16.021404 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:16.021459 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.021412 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:16.021555 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.021527 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:16.021648 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:16.021627 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:17.150317 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150638 2579 generic.go:356] "Generic (PLEG): container finished" podID="fe747a0c-5a1d-4db3-876a-f26e1852450e" containerID="d9ef7366120b4c4fc283f770357906066214f7858437e278aa71d96becebba74" exitCode=1 Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150683 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"7cd9f2d198fc98833880be6c65d2dd2fb87010e6e1d1bedda02ef05ba9592668"} Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150760 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"f10ec3d1d2c100634d9d377af4e9631d2f789b709b16f99c78e1ec8fa7c45f4b"} Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150790 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"7a32cb5336baf323eba10b506b016b4435494b60398073018e5abad61534cd79"} Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150818 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"063dd44e0d31c743b9fe74bc8ff72c7068ee74751a532bf17c5ab2e3b8cd163d"} Nov 03 17:33:17.150844 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150844 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerDied","Data":"d9ef7366120b4c4fc283f770357906066214f7858437e278aa71d96becebba74"} Nov 03 17:33:17.151016 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:17.150873 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"2c1eeea5e481e7930d051daab834e6120aaf472d31bd5cb51067d381ebef9831"} Nov 03 17:33:18.021874 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:18.021689 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:18.022040 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:18.021687 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:18.022040 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:18.021961 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:18.022040 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:18.021737 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:18.022128 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:18.022041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:18.022128 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:18.022110 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:18.155564 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:18.155533 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="676bd42e14226ec68974e117496da1f085043c2e2bdf3040c470dd146d432d55" exitCode=0 Nov 03 17:33:18.155914 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:18.155592 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"676bd42e14226ec68974e117496da1f085043c2e2bdf3040c470dd146d432d55"} Nov 03 17:33:19.161147 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:19.161125 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:19.161821 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:19.161606 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"e2fa4c099894d0a8e9565462080d95fb666c110b396396ee3429ce68692d1b4b"} Nov 03 17:33:19.164819 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:19.164797 2579 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="7fcf49cd20b833bc2385765e3d30f6195e9c633ab5cb46bdcbc9062cecf8881c" exitCode=0 Nov 03 17:33:19.164907 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:19.164845 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerDied","Data":"7fcf49cd20b833bc2385765e3d30f6195e9c633ab5cb46bdcbc9062cecf8881c"} Nov 03 17:33:20.021299 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:20.021267 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:20.021435 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:20.021306 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:20.021435 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:20.021370 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:20.021435 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:20.021386 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:20.021547 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:20.021438 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:20.021547 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:20.021503 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:20.170646 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:20.170619 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" event={"ID":"92dc67cc-a063-43af-9a22-432e61ee10f4","Type":"ContainerStarted","Data":"6e177d795999f8600f6164f80ecb871dfd1afd9933f119817eee22dded45d6eb"} Nov 03 17:33:20.318307 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:20.318219 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xmdk5" podStartSLOduration=8.537194283 podStartE2EDuration="21.318205294s" totalImagesPullingTime="12.781011011s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:20.317751934 +0000 UTC m=+21.807254092" watchObservedRunningTime="2026-11-03 17:33:20.318205294 +0000 UTC m=+21.807707472" Nov 03 17:33:22.021777 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.021742 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:22.021777 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.021770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:22.022672 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.021774 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:22.022672 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:22.021861 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:22.022672 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:22.021928 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:22.022672 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:22.021989 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:22.178358 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.178327 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:22.178696 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.178672 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"87e70c88e64c4820e270188c0a32be38d7ef0f08d3e0327b9f853fc568566523"} Nov 03 17:33:22.178950 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.178931 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:22.179000 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.178992 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:22.179143 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.179129 2579 scope.go:122] "RemoveContainer" containerID="d9ef7366120b4c4fc283f770357906066214f7858437e278aa71d96becebba74" Nov 03 17:33:22.196685 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:22.196514 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:23.185001 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:23.184975 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:23.185419 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:23.185348 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" event={"ID":"fe747a0c-5a1d-4db3-876a-f26e1852450e","Type":"ContainerStarted","Data":"1dc9952b33eed58be28c1ee96e9490c34f36fa0aa9b302f53ba30b4cc8afe59a"} Nov 03 17:33:23.185621 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:23.185602 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:23.201504 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:23.201477 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:23.367581 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:23.367531 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" podStartSLOduration=9.375011771 podStartE2EDuration="24.367517286s" totalImagesPullingTime="14.992505515s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:23.367008967 +0000 UTC m=+24.856511170" watchObservedRunningTime="2026-11-03 17:33:23.367517286 +0000 UTC m=+24.857019496" Nov 03 17:33:24.007091 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.007055 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fnq58"] Nov 03 17:33:24.007257 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.007203 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:24.007335 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:24.007313 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:24.009336 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.009313 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xp42v"] Nov 03 17:33:24.009464 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.009406 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:24.009525 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:24.009473 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:24.012938 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.012909 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tlq8b"] Nov 03 17:33:24.013050 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:24.013012 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:24.013104 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:24.013082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:26.021687 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:26.021660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:26.022134 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:26.021660 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:26.022134 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:26.021762 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:26.022134 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:26.021785 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:26.022134 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:26.021846 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:26.022134 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:26.021918 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:28.021792 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.021762 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:28.022220 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.021857 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:28.022220 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.021865 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:28.022220 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:28.021858 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-fnq58" podUID="fedfb5ca-94b5-4850-8020-a7279aa49eae" Nov 03 17:33:28.022220 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:28.021954 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-xp42v" podUID="333b0df1-e4ba-4064-979c-655f7e52b9f0" Nov 03 17:33:28.022220 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:28.022022 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tlq8b" podUID="cbea6928-9579-4606-afd8-42fccd905154" Nov 03 17:33:28.801278 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.801250 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeReady" Nov 03 17:33:28.801406 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.801385 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Nov 03 17:33:28.917104 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.917070 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5ndm"] Nov 03 17:33:28.932990 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.932961 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tgbt"] Nov 03 17:33:28.933126 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.933080 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:28.950927 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.950897 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:28.958509 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.958480 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4x9kc"] Nov 03 17:33:28.978462 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.978436 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p4zh5"] Nov 03 17:33:28.978622 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.978541 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:28.999808 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.999644 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gcgd8"] Nov 03 17:33:28.999921 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:28.999754 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:29.016379 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.016351 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwc8l\" (UniqueName: \"kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l\") pod \"kas-connection-checker-685895c7c9-6tgbt\" (UID: \"dfad8e5c-4000-47b2-b39c-ea805fd6e315\") " pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:29.016482 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.016463 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-999ff\" (UniqueName: \"kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff\") pod \"kas-connection-checker-685895c7c9-w5ndm\" (UID: \"defb7da5-8103-40ef-9607-49c56c281187\") " pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:29.016654 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.016642 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.016712 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.016698 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.022300 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.022278 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.026229 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.026208 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h"] Nov 03 17:33:29.038010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.037984 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 17:33:29.038154 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.038136 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-htgnv\"" Nov 03 17:33:29.039786 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.039761 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn"] Nov 03 17:33:29.040513 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.040484 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:29.054960 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.054939 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-5b97b666bc-r5q4p"] Nov 03 17:33:29.055086 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.055069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.077837 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.077800 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:33:29.077944 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.077806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.091815 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.091789 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8"] Nov 03 17:33:29.091914 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.091886 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.095584 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.095541 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.095795 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.095770 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 17:33:29.095940 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.095920 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 17:33:29.096007 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.095967 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.096152 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.096136 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.096970 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.096707 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.106513 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.106483 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj"] Nov 03 17:33:29.106675 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.106652 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.116990 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.116969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72r2f\" (UniqueName: \"kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.117078 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.116999 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.117078 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117020 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.117078 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc8l\" (UniqueName: \"kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l\") pod \"kas-connection-checker-685895c7c9-6tgbt\" (UID: \"dfad8e5c-4000-47b2-b39c-ea805fd6e315\") " pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:29.117078 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117066 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.117203 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117141 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xddlm\" (UniqueName: \"kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.117203 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117159 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.117203 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117191 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.117291 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117247 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.117324 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117296 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999ff\" (UniqueName: \"kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff\") pod \"kas-connection-checker-685895c7c9-w5ndm\" (UID: \"defb7da5-8103-40ef-9607-49c56c281187\") " pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:29.117355 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117326 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v88mt\" (UniqueName: \"kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt\") pod \"volume-data-source-validator-58c9f5f46d-dfx2h\" (UID: \"5d987bf9-3048-4b49-ba64-010106066297\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:29.117355 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117349 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8zdt\" (UniqueName: \"kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt\") pod \"kas-connection-checker-685895c7c9-p4zh5\" (UID: \"bea67a96-85b1-41d3-b177-bfec1d287457\") " pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:29.117414 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117380 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.117414 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117397 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.117505 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.117489 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.120712 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.120698 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mpbwn"] Nov 03 17:33:29.120836 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.120807 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.130542 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.130518 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 17:33:29.137189 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.137164 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:33:29.137288 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.137270 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.138178 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138160 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 17:33:29.138277 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138219 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 17:33:29.138387 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138372 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 17:33:29.138454 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138414 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 17:33:29.138591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138559 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9ftgc\"" Nov 03 17:33:29.138635 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138612 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-55gxq\"" Nov 03 17:33:29.138635 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138629 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vrg7g\"" Nov 03 17:33:29.138706 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138651 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 17:33:29.138742 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.138712 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mnwfr\"" Nov 03 17:33:29.147559 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.147543 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc"] Nov 03 17:33:29.147668 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.147654 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.164083 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.164058 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln"] Nov 03 17:33:29.164215 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.164197 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.176515 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.176498 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-chggk"] Nov 03 17:33:29.176623 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.176608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:29.181920 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.181898 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc8l\" (UniqueName: \"kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l\") pod \"kas-connection-checker-685895c7c9-6tgbt\" (UID: \"dfad8e5c-4000-47b2-b39c-ea805fd6e315\") " pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:29.181920 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.181917 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999ff\" (UniqueName: \"kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff\") pod \"kas-connection-checker-685895c7c9-w5ndm\" (UID: \"defb7da5-8103-40ef-9607-49c56c281187\") " pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:29.190279 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.190259 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rtv8j"] Nov 03 17:33:29.190483 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.190466 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.191397 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.191070 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.194798 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.193297 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 17:33:29.195030 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195009 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 17:33:29.195090 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195064 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.195258 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195227 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.195448 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195432 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.195448 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195439 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.195604 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195466 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.195604 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195485 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 17:33:29.195688 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195623 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.195727 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195715 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.195767 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195734 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.195815 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195796 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 17:33:29.195880 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195862 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.195948 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.195876 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 17:33:29.198361 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.198345 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 17:33:29.199061 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.199047 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 17:33:29.206716 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206699 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5ndm"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206729 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tgbt"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206744 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4x9kc"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206757 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p4zh5"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206772 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206785 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gcgd8"] Nov 03 17:33:29.206801 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206798 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b97b666bc-r5q4p"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206800 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206811 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206824 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mpbwn"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206840 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206853 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206865 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chggk"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206881 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206895 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206910 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rtv8j"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206923 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8"] Nov 03 17:33:29.207009 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.206937 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:33:29.218825 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218803 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72r2f\" (UniqueName: \"kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.218927 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218835 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.218927 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.218927 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218871 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.218927 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.219156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218947 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjp2c\" (UniqueName: \"kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.219156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.218997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.219156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219035 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.219156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219055 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dm74\" (UniqueName: \"kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.219156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219153 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2qk9\" (UniqueName: \"kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219219 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219249 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219277 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219301 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.219332 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219342 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhgmr\" (UniqueName: \"kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.219405 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.219387 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.719375477 +0000 UTC m=+31.208877633 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219497 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219588 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219618 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219647 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219668 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zdt\" (UniqueName: \"kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt\") pod \"kas-connection-checker-685895c7c9-p4zh5\" (UID: \"bea67a96-85b1-41d3-b177-bfec1d287457\") " pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219700 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219775 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.219788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219778 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219820 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219839 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219891 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.219925 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220034 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.220093 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d69bj\" (UniqueName: \"kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220109 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220132 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220158 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220190 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220221 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddlm\" (UniqueName: \"kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220254 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220308 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220289 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220343 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220402 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220434 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88mt\" (UniqueName: \"kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt\") pod \"volume-data-source-validator-58c9f5f46d-dfx2h\" (UID: \"5d987bf9-3048-4b49-ba64-010106066297\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220461 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220506 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220555 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220601 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220632 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220659 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.220745 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.220685 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.221362 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.221339 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.221944 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.221884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.222035 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.222002 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.222319 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.222297 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.222456 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.222442 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.223910 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.223895 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.239055 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239000 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 17:33:29.239210 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239196 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 17:33:29.239285 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239212 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 17:33:29.239285 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239274 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-42p6c\"" Nov 03 17:33:29.239398 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239378 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 17:33:29.239461 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239405 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 17:33:29.239461 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239442 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 17:33:29.239555 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239461 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-79kzw\"" Nov 03 17:33:29.239627 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239558 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hndzv\"" Nov 03 17:33:29.239627 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239561 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 17:33:29.239627 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239592 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n729t\"" Nov 03 17:33:29.239627 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239586 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jhqc8\"" Nov 03 17:33:29.239627 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.239604 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr55j\"" Nov 03 17:33:29.244534 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.244515 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\"" Nov 03 17:33:29.248156 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.248141 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:33:29.260014 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.259961 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:29.260299 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.260282 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\"" Nov 03 17:33:29.260362 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.260345 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:33:29.282404 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.282377 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zdt\" (UniqueName: \"kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt\") pod \"kas-connection-checker-685895c7c9-p4zh5\" (UID: \"bea67a96-85b1-41d3-b177-bfec1d287457\") " pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:29.282763 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.282744 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72r2f\" (UniqueName: \"kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.283908 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.283880 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddlm\" (UniqueName: \"kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.284450 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.284432 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88mt\" (UniqueName: \"kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt\") pod \"volume-data-source-validator-58c9f5f46d-dfx2h\" (UID: \"5d987bf9-3048-4b49-ba64-010106066297\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:29.286254 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.286232 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:29.290939 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290907 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.290939 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290925 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.291111 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290912 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:33:29.291111 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290910 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.291111 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290968 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 17:33:29.291111 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.290980 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:33:29.291111 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.291030 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.291250 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.291120 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:29.315323 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.314655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322177 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322199 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322240 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322269 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322291 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp2c\" (UniqueName: \"kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322318 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322340 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm74\" (UniqueName: \"kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322365 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qk9\" (UniqueName: \"kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322465 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322492 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322522 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgmr\" (UniqueName: \"kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.323005 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322557 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322605 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322629 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322656 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322687 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322741 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322772 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322845 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322873 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322906 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69bj\" (UniqueName: \"kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322909 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322936 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322987 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.323006 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:29.323838 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323015 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpb8d\" (UniqueName: \"kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7682\" (UniqueName: \"kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682\") pod \"network-check-source-6f78d59f99-sqcln\" (UID: \"8aa2379e-1eed-4cce-a9df-e96665bf3fd9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323112 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323140 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323173 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.322911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323208 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323236 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpwnq\" (UniqueName: \"kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.323020 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.323262 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.323473 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.823460002 +0000 UTC m=+31.312962162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.323698 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.823686078 +0000 UTC m=+31.313188237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323713 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323740 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.324642 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.323974 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.324637 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.324675 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.324710 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.324759 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.824745084 +0000 UTC m=+31.314247262 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.324773 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.324789 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.325263 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.325301 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.825290702 +0000 UTC m=+31.314792871 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:29.325394 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.325324 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.825318585 +0000 UTC m=+31.314820755 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:29.325889 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.325506 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.325889 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.325759 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.325997 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.325924 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.326378 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.326351 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.326483 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.326355 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:29.326483 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.326408 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:29.326483 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.326449 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.826435834 +0000 UTC m=+31.315938004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:29.327042 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.326736 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.327452 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.327426 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.328200 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.329885 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330081 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330105 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.330334 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330113 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.330653 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.330653 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.330608 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:29.370459 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.370425 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:29.389926 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.384695 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:29.396715 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.396564 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm74\" (UniqueName: \"kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.397443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.397159 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp2c\" (UniqueName: \"kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.397443 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.397391 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.398677 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.398201 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69bj\" (UniqueName: \"kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.398677 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.398389 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.405721 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.404260 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgmr\" (UniqueName: \"kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.405721 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.405596 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qk9\" (UniqueName: \"kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.406239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.406198 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.408644 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.408601 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.427364 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.427300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.428607 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.428543 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.429776 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.429709 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.429934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpb8d\" (UniqueName: \"kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.429999 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7682\" (UniqueName: \"kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682\") pod \"network-check-source-6f78d59f99-sqcln\" (UID: \"8aa2379e-1eed-4cce-a9df-e96665bf3fd9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.430044 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.430093 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwnq\" (UniqueName: \"kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.430203 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.430616 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.430458 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.431304 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.431059 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:29.431304 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.431145 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:29.931122283 +0000 UTC m=+31.420624453 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:29.431932 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.431911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.464986 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.464933 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:29.486880 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.486851 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:29.494023 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.494001 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpb8d\" (UniqueName: \"kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.497073 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.497043 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwnq\" (UniqueName: \"kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.497556 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.497534 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7682\" (UniqueName: \"kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682\") pod \"network-check-source-6f78d59f99-sqcln\" (UID: \"8aa2379e-1eed-4cce-a9df-e96665bf3fd9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:29.502338 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.502317 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:29.547799 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.547764 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:29.696827 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.693795 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-6tgbt"] Nov 03 17:33:29.697805 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.697431 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-w5ndm"] Nov 03 17:33:29.698859 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.698834 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfad8e5c_4000_47b2_b39c_ea805fd6e315.slice/crio-0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a WatchSource:0}: Error finding container 0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a: Status 404 returned error can't find the container with id 0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a Nov 03 17:33:29.700040 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.700007 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddefb7da5_8103_40ef_9607_49c56c281187.slice/crio-17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f WatchSource:0}: Error finding container 17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f: Status 404 returned error can't find the container with id 17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f Nov 03 17:33:29.714918 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.714890 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-7c4986d449-4x9kc"] Nov 03 17:33:29.717935 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.717906 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cf9464_6022_41a2_b0bc_be87d6b3b90e.slice/crio-3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89 WatchSource:0}: Error finding container 3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89: Status 404 returned error can't find the container with id 3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89 Nov 03 17:33:29.734538 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.734509 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:29.734717 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.734636 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:29.734717 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.734712 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.734692625 +0000 UTC m=+32.224194800 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:29.756933 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.756904 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/kas-connection-checker-685895c7c9-p4zh5"] Nov 03 17:33:29.760608 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.760556 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea67a96_85b1_41d3_b177_bfec1d287457.slice/crio-fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188 WatchSource:0}: Error finding container fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188: Status 404 returned error can't find the container with id fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188 Nov 03 17:33:29.786786 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.786556 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h"] Nov 03 17:33:29.792078 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.792045 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d987bf9_3048_4b49_ba64_010106066297.slice/crio-7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf WatchSource:0}: Error finding container 7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf: Status 404 returned error can't find the container with id 7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf Nov 03 17:33:29.803657 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.803563 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn"] Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.835806 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.835911 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.835949 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.835979 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.836010 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.836077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836227 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836209242 +0000 UTC m=+32.325711414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836309 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836346 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836334937 +0000 UTC m=+32.325837110 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836398 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836434 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836422498 +0000 UTC m=+32.325924670 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836489 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836519 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836510491 +0000 UTC m=+32.326012655 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836591 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:29.836743 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836601 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:29.837647 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836629 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836620363 +0000 UTC m=+32.326122534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:29.837647 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836682 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:29.837647 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836692 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:29.837647 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.836720 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.836710003 +0000 UTC m=+32.326212173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:29.858531 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.858317 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7cd56b8dbc-mpbwn"] Nov 03 17:33:29.902612 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.902586 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc"] Nov 03 17:33:29.905694 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.905672 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a59ced0_24a7_45f0_a0c7_de168cbc59a7.slice/crio-b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04 WatchSource:0}: Error finding container b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04: Status 404 returned error can't find the container with id b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04 Nov 03 17:33:29.908058 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.908033 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln"] Nov 03 17:33:29.911030 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.911010 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2379e_1eed_4cce_a9df_e96665bf3fd9.slice/crio-86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c WatchSource:0}: Error finding container 86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c: Status 404 returned error can't find the container with id 86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c Nov 03 17:33:29.937456 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.937435 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:29.937603 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.937564 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:29.937661 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:29.937639 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:30.937625207 +0000 UTC m=+32.427127378 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:29.963939 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:29.963915 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rtv8j"] Nov 03 17:33:29.966318 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:29.966296 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67333be4_e8fc_47bd_89f3_5873637f8193.slice/crio-cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140 WatchSource:0}: Error finding container cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140: Status 404 returned error can't find the container with id cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140 Nov 03 17:33:30.021048 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.021019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:30.021048 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.021050 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:30.021256 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.021241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:30.075924 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.075894 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:33:30.076447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.076087 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\"" Nov 03 17:33:30.076447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.076096 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:33:30.076447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.076154 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\"" Nov 03 17:33:30.076447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.076153 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\"" Nov 03 17:33:30.203253 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.203225 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" event={"ID":"dfad8e5c-4000-47b2-b39c-ea805fd6e315","Type":"ContainerStarted","Data":"fa942526dee02da94af5c874dd68d379ee06be090bbf6012bea5bac07c85d042"} Nov 03 17:33:30.203386 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.203285 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" event={"ID":"dfad8e5c-4000-47b2-b39c-ea805fd6e315","Type":"ContainerStarted","Data":"0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a"} Nov 03 17:33:30.204867 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.204842 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" event={"ID":"defb7da5-8103-40ef-9607-49c56c281187","Type":"ContainerStarted","Data":"867d219af3ee1798ddd247a09687a53f4ae2200df5269f2f94144f9afcd49d67"} Nov 03 17:33:30.204991 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.204891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" event={"ID":"defb7da5-8103-40ef-9607-49c56c281187","Type":"ContainerStarted","Data":"17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f"} Nov 03 17:33:30.206122 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.206103 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerStarted","Data":"3cf4b0281d310179605867818704d02af998c06f6a52e487b3cd4df1e73e13ea"} Nov 03 17:33:30.207381 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.207360 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" event={"ID":"5d987bf9-3048-4b49-ba64-010106066297","Type":"ContainerStarted","Data":"7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf"} Nov 03 17:33:30.208512 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.208494 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" event={"ID":"8aa2379e-1eed-4cce-a9df-e96665bf3fd9","Type":"ContainerStarted","Data":"86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c"} Nov 03 17:33:30.209651 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.209624 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" event={"ID":"0a59ced0-24a7-45f0-a0c7-de168cbc59a7","Type":"ContainerStarted","Data":"b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04"} Nov 03 17:33:30.210721 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.210704 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" event={"ID":"9584d3a2-671c-48e3-89f9-dfd8e38db4ab","Type":"ContainerStarted","Data":"02132f883a1d7004bbf2b5332eb6b5e468781411a4d1fcaa5a03c7961053c193"} Nov 03 17:33:30.211869 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.211844 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" event={"ID":"56cf9464-6022-41a2-b0bc-be87d6b3b90e","Type":"ContainerStarted","Data":"3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89"} Nov 03 17:33:30.212961 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.212935 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rtv8j" event={"ID":"67333be4-e8fc-47bd-89f3-5873637f8193","Type":"ContainerStarted","Data":"cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140"} Nov 03 17:33:30.214517 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.214496 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" event={"ID":"bea67a96-85b1-41d3-b177-bfec1d287457","Type":"ContainerStarted","Data":"5450922b3c701ddd8c12d400372a8173f695d7da1411b5fce43b78bf0819bcf1"} Nov 03 17:33:30.214630 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.214534 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" event={"ID":"bea67a96-85b1-41d3-b177-bfec1d287457","Type":"ContainerStarted","Data":"fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188"} Nov 03 17:33:30.744823 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.744782 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:30.745002 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.744947 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:30.745062 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.745027 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.745004312 +0000 UTC m=+34.234506483 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:30.845905 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.845865 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:30.846070 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.845962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:30.846070 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.846030 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:30.846163 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.846118 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:30.846163 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.846154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:30.846245 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.846181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:30.846334 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846314 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:30.846428 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846336 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:30.846428 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846399 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846378884 +0000 UTC m=+34.335881056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:30.846541 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846475 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:30.846541 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846487 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:30.846541 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846523 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846512037 +0000 UTC m=+34.336014206 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:30.846742 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846613 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846602714 +0000 UTC m=+34.336104887 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:30.846742 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846677 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:30.846742 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846708 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846698487 +0000 UTC m=+34.336200658 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:30.846903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846754 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:30.846903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846781 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846771936 +0000 UTC m=+34.336274107 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:30.846903 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846860 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:30.847042 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.846932 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.846914014 +0000 UTC m=+34.336416174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:30.947497 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:30.947455 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:30.947671 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.947613 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:30.947748 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:30.947674 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:32.947655053 +0000 UTC m=+34.437157222 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:31.165857 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.165173 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" podStartSLOduration=351.165154066 podStartE2EDuration="5m51.165154066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:31.163386635 +0000 UTC m=+32.652888814" watchObservedRunningTime="2026-11-03 17:33:31.165154066 +0000 UTC m=+32.654656246" Nov 03 17:33:31.285966 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.285904 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" podStartSLOduration=351.285884692 podStartE2EDuration="5m51.285884692s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:31.283894213 +0000 UTC m=+32.773396383" watchObservedRunningTime="2026-11-03 17:33:31.285884692 +0000 UTC m=+32.775386871" Nov 03 17:33:31.404302 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.404138 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" podStartSLOduration=351.404117726 podStartE2EDuration="5m51.404117726s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:27:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:31.401595174 +0000 UTC m=+32.891097353" watchObservedRunningTime="2026-11-03 17:33:31.404117726 +0000 UTC m=+32.893619907" Nov 03 17:33:31.961613 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.961059 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:31.961613 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.961174 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:31.961613 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:31.961319 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 17:33:31.961613 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:31.961379 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:34:03.961361354 +0000 UTC m=+65.450863534 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : secret "metrics-daemon-secret" not found Nov 03 17:33:31.980628 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:31.980555 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:32.063299 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.062747 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:32.066177 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.066123 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:32.182239 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.181980 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:32.200385 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.200288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:32.623391 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.623253 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-xp42v"] Nov 03 17:33:32.672135 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.672102 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-fnq58"] Nov 03 17:33:32.768533 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.768493 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:32.768742 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.768728 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:32.768800 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.768785 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.768771982 +0000 UTC m=+38.258274152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:32.789706 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:32.789666 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod333b0df1_e4ba_4064_979c_655f7e52b9f0.slice/crio-524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc WatchSource:0}: Error finding container 524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc: Status 404 returned error can't find the container with id 524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.869758 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.869852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.869896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.869927 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.869958 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.870020 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870179 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.870160235 +0000 UTC m=+38.359662395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870269 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870307 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.870293973 +0000 UTC m=+38.359796130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870364 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870395 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.870385621 +0000 UTC m=+38.359887782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870488 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.870521 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.87051078 +0000 UTC m=+38.360012943 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.871483 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:32.871591 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.871499 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:32.872533 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.871550 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.871534089 +0000 UTC m=+38.361036246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:32.873563 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.873436 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:32.873563 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.873452 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:32.873563 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.873500 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.873485875 +0000 UTC m=+38.362988034 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:32.970878 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:32.970611 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:32.971443 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.971268 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:32.971443 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:32.971360 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:36.971338881 +0000 UTC m=+38.460841050 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:33.230517 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:33.230449 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xp42v" event={"ID":"333b0df1-e4ba-4064-979c-655f7e52b9f0","Type":"ContainerStarted","Data":"524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc"} Nov 03 17:33:33.233022 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:33.232992 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" event={"ID":"5d987bf9-3048-4b49-ba64-010106066297","Type":"ContainerStarted","Data":"5e5750b8ba484cf2aa2f3490fbcb3bb838198819202beaa7d3211d9491f6003a"} Nov 03 17:33:33.236142 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:33.236116 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-fnq58" event={"ID":"fedfb5ca-94b5-4850-8020-a7279aa49eae","Type":"ContainerStarted","Data":"6f19982762c0d5067082948ddff301b09e190afa7498f2f71a2015571a776895"} Nov 03 17:33:35.246608 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.246518 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerStarted","Data":"679089a3b1b67cbbc345ce13f64f0b9e7f3c5ca172c1a38e64b59b296f99352f"} Nov 03 17:33:35.247075 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.246808 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:35.248296 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.248049 2579 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-mpbwn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Nov 03 17:33:35.248296 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.248097 2579 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Nov 03 17:33:35.253756 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.253729 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" event={"ID":"56cf9464-6022-41a2-b0bc-be87d6b3b90e","Type":"ContainerStarted","Data":"2494de23c64b3d041fe3eab953f56af62de082366fbd4861b6bd0800d9a351d9"} Nov 03 17:33:35.367360 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.367305 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" podStartSLOduration=303.350683142 podStartE2EDuration="5m6.367287066s" totalImagesPullingTime="3.016603924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:33.35265541 +0000 UTC m=+34.842157591" watchObservedRunningTime="2026-11-03 17:33:35.367287066 +0000 UTC m=+36.856789244" Nov 03 17:33:35.491173 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.489988 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podStartSLOduration=305.391170012 podStartE2EDuration="5m10.489967522s" totalImagesPullingTime="5.09879751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:35.368549667 +0000 UTC m=+36.858051845" watchObservedRunningTime="2026-11-03 17:33:35.489967522 +0000 UTC m=+36.979469701" Nov 03 17:33:35.491173 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:35.490943 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" podStartSLOduration=302.255328106 podStartE2EDuration="5m7.490933634s" totalImagesPullingTime="5.235605528s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:35.488759045 +0000 UTC m=+36.978261226" watchObservedRunningTime="2026-11-03 17:33:35.490933634 +0000 UTC m=+36.980435813" Nov 03 17:33:36.261507 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.260503 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/0.log" Nov 03 17:33:36.261507 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.260547 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerID="679089a3b1b67cbbc345ce13f64f0b9e7f3c5ca172c1a38e64b59b296f99352f" exitCode=255 Nov 03 17:33:36.261507 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.260667 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerDied","Data":"679089a3b1b67cbbc345ce13f64f0b9e7f3c5ca172c1a38e64b59b296f99352f"} Nov 03 17:33:36.261507 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.261091 2579 scope.go:122] "RemoveContainer" containerID="679089a3b1b67cbbc345ce13f64f0b9e7f3c5ca172c1a38e64b59b296f99352f" Nov 03 17:33:36.266367 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.265689 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" event={"ID":"0a59ced0-24a7-45f0-a0c7-de168cbc59a7","Type":"ContainerStarted","Data":"892c11727184de612844e707852498c38ff5761c5e2198c6e2930072deebd5c0"} Nov 03 17:33:36.268220 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.268196 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" event={"ID":"9584d3a2-671c-48e3-89f9-dfd8e38db4ab","Type":"ContainerStarted","Data":"e7059d6ee070d5ae522c80169fcc30b8f9d2e82d503bf94118c968f497868341"} Nov 03 17:33:36.280657 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.279208 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-42zww"] Nov 03 17:33:36.313007 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.312426 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.351079 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.351042 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42zww"] Nov 03 17:33:36.365957 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.365790 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:33:36.365957 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.365887 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\"" Nov 03 17:33:36.365957 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.365936 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:33:36.410194 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.410161 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.410301 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.410205 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.410415 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.410399 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zc79m\" (UniqueName: \"kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.413404 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.410480 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.414686 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.413624 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.508777 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.508719 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" podStartSLOduration=304.886243425 podStartE2EDuration="5m10.508704503s" totalImagesPullingTime="5.622461078s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:36.507893012 +0000 UTC m=+37.997395191" watchObservedRunningTime="2026-11-03 17:33:36.508704503 +0000 UTC m=+37.998206682" Nov 03 17:33:36.514333 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514275 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.514447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514373 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.514447 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514402 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.514545 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.514444 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:36.514545 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514496 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc79m\" (UniqueName: \"kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.514545 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.514520 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:37.014501936 +0000 UTC m=+38.504004109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:36.514702 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514587 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.514702 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.514662 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:36.514702 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.514690 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:37.014681348 +0000 UTC m=+38.504183517 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:36.514876 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.514862 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.515079 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.515052 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.577839 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.577811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc79m\" (UniqueName: \"kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:36.628226 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.628154 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" podStartSLOduration=301.953285881 podStartE2EDuration="5m7.628134632s" totalImagesPullingTime="5.674848751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:36.62704152 +0000 UTC m=+38.116543701" watchObservedRunningTime="2026-11-03 17:33:36.628134632 +0000 UTC m=+38.117636812" Nov 03 17:33:36.817656 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.817552 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:36.817811 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.817678 2579 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:36.817811 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.817751 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.8177336 +0000 UTC m=+46.307235773 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:36.918584 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918546 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:36.918753 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918655 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:36.918753 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918692 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:36.918753 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918712 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:36.918753 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918714 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918773 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.91875259 +0000 UTC m=+46.408254749 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918808 2579 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918853 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918884 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918893 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.918877872 +0000 UTC m=+46.408380048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:36.918928 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:36.918963 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918953 2579 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.918989 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.918977534 +0000 UTC m=+46.408479721 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919034 2579 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919047 2579 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919051 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.919041698 +0000 UTC m=+46.408543861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919038 2579 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919081 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.919070197 +0000 UTC m=+46.408572371 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:36.919234 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:36.919101 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.919089396 +0000 UTC m=+46.408591605 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:37.019808 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.019780 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:37.020011 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.019857 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:37.020011 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.019896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:37.020011 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.019931 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:37.020011 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.019999 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:38.01997844 +0000 UTC m=+39.509480603 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:37.020222 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.020071 2579 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:37.020222 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.020120 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.020101805 +0000 UTC m=+46.509603983 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:37.020222 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.020154 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:37.020222 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.020192 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:38.020181002 +0000 UTC m=+39.509683162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:37.276974 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.276942 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:37.277415 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.277400 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/0.log" Nov 03 17:33:37.277459 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.277430 2579 generic.go:356] "Generic (PLEG): container finished" podID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" exitCode=255 Nov 03 17:33:37.277515 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.277499 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerDied","Data":"16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0"} Nov 03 17:33:37.277587 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.277556 2579 scope.go:122] "RemoveContainer" containerID="679089a3b1b67cbbc345ce13f64f0b9e7f3c5ca172c1a38e64b59b296f99352f" Nov 03 17:33:37.278010 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.277987 2579 scope.go:122] "RemoveContainer" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" Nov 03 17:33:37.278239 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:37.278208 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:37.280338 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.280313 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" event={"ID":"8aa2379e-1eed-4cce-a9df-e96665bf3fd9","Type":"ContainerStarted","Data":"6fac5f747925d32892120e4f72f0dbe933674ea0d62af3ecb3f9f4501c9ee869"} Nov 03 17:33:37.282752 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.282730 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rtv8j" event={"ID":"67333be4-e8fc-47bd-89f3-5873637f8193","Type":"ContainerStarted","Data":"204fbf3c0e5493e4dfa2f46a9d5f5ba7f7a46bb661b8556a7b992550b0dcd0a9"} Nov 03 17:33:37.284495 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.284469 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-xp42v" event={"ID":"333b0df1-e4ba-4064-979c-655f7e52b9f0","Type":"ContainerStarted","Data":"5405893ed06d3a740b7d8ca1b9141b382ba9d5d85ecaa68a41c68bbbf3025645"} Nov 03 17:33:37.382278 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.382212 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7"] Nov 03 17:33:37.412852 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.412822 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:37.445193 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.445168 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7"] Nov 03 17:33:37.525504 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.525466 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d499d\" (UniqueName: \"kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d\") pod \"migrator-66df5d496b-r2zn7\" (UID: \"cb176382-bf16-4088-9f2e-0ec9932511f5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:37.535612 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.535539 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rtv8j" podStartSLOduration=3.231563384 podStartE2EDuration="9.535525619s" totalImagesPullingTime="6.303962235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:33:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:37.534889872 +0000 UTC m=+39.024392049" watchObservedRunningTime="2026-11-03 17:33:37.535525619 +0000 UTC m=+39.025027797" Nov 03 17:33:37.553834 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.553791 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:37.553834 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.553813 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 17:33:37.554060 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.554040 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-df4kg\"" Nov 03 17:33:37.626612 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.626558 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d499d\" (UniqueName: \"kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d\") pod \"migrator-66df5d496b-r2zn7\" (UID: \"cb176382-bf16-4088-9f2e-0ec9932511f5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:37.656594 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.656480 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" podStartSLOduration=298.550795786 podStartE2EDuration="5m5.656466623s" totalImagesPullingTime="7.105670837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:28:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:37.656085984 +0000 UTC m=+39.145588163" watchObservedRunningTime="2026-11-03 17:33:37.656466623 +0000 UTC m=+39.145968800" Nov 03 17:33:37.689353 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.689325 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d499d\" (UniqueName: \"kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d\") pod \"migrator-66df5d496b-r2zn7\" (UID: \"cb176382-bf16-4088-9f2e-0ec9932511f5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:37.737225 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.737185 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:37.775518 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:37.774745 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-xp42v" podStartSLOduration=34.576379172 podStartE2EDuration="38.77472606s" totalImagesPullingTime="4.198346888s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:32:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:33:37.773175749 +0000 UTC m=+39.262677927" watchObservedRunningTime="2026-11-03 17:33:37.77472606 +0000 UTC m=+39.264228245" Nov 03 17:33:38.030350 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.030305 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:38.030538 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.030382 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:38.030538 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:38.030447 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:38.030538 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:38.030528 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:40.030507796 +0000 UTC m=+41.520009965 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:38.030760 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:38.030566 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:38.030760 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:38.030642 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:40.030631027 +0000 UTC m=+41.520133199 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:38.122452 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.122419 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7"] Nov 03 17:33:38.125777 ip-10-0-139-234 kubenswrapper[2579]: W1103 17:33:38.125748 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb176382_bf16_4088_9f2e_0ec9932511f5.slice/crio-883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865 WatchSource:0}: Error finding container 883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865: Status 404 returned error can't find the container with id 883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865 Nov 03 17:33:38.289805 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.289719 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" event={"ID":"cb176382-bf16-4088-9f2e-0ec9932511f5","Type":"ContainerStarted","Data":"883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865"} Nov 03 17:33:38.291545 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.291521 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:38.291788 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.291764 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:38.291907 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:38.291888 2579 scope.go:122] "RemoveContainer" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" Nov 03 17:33:38.292108 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:38.292080 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:39.465395 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:39.465355 2579 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:39.465869 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:39.465784 2579 scope.go:122] "RemoveContainer" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" Nov 03 17:33:39.466000 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:39.465977 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:40.051804 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:40.051769 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:40.051973 ip-10-0-139-234 kubenswrapper[2579]: I1103 17:33:40.051821 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:40.051973 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:40.051921 2579 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:40.051973 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:40.051920 2579 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:40.051973 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:40.051967 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.051955129 +0000 UTC m=+45.541457299 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:40.052118 ip-10-0-139-234 kubenswrapper[2579]: E1103 17:33:40.051993 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:44.051974571 +0000 UTC m=+45.541476742 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:40.112698 ip-10-0-139-234 systemd[1]: Stopping Kubernetes Kubelet... Nov 03 17:33:40.120088 ip-10-0-139-234 systemd[1]: kubelet.service: Deactivated successfully. Nov 03 17:33:40.120260 ip-10-0-139-234 systemd[1]: Stopped Kubernetes Kubelet. Nov 03 17:33:40.120429 ip-10-0-139-234 systemd[1]: kubelet.service: Consumed 3.106s CPU time. Nov 03 17:33:40.132188 ip-10-0-139-234 systemd[1]: Starting Kubernetes Kubelet... Nov 03 17:33:40.200532 ip-10-0-139-234 kubenswrapper[4980]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:33:40.200532 ip-10-0-139-234 kubenswrapper[4980]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Nov 03 17:33:40.200532 ip-10-0-139-234 kubenswrapper[4980]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:33:40.200532 ip-10-0-139-234 kubenswrapper[4980]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Nov 03 17:33:40.202615 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202523 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.202615 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202613 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.202615 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202617 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202622 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202627 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202630 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202633 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202636 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202638 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202641 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202644 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202646 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202649 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202652 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202655 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202657 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202660 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202663 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202665 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202668 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202670 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202673 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.202722 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202675 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202678 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202684 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202687 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202689 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202692 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202695 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202699 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202704 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202708 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202710 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202713 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202717 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202720 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202722 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202725 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202728 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202731 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202734 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202737 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.203262 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202739 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202742 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202744 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202747 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202750 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202752 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202755 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202757 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202760 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202762 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202766 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202769 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202771 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202774 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202776 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202779 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202782 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202785 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202787 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.203818 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202790 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202793 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202795 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202798 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202800 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202803 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202806 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202809 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202811 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202814 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202816 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202819 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202822 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202824 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202827 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202829 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202832 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202835 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202837 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202839 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.204327 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202842 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202845 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202848 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202850 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202853 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202855 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202858 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202860 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202863 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202865 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202868 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202871 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202873 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202876 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202878 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202882 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202885 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202889 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202891 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.204862 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202894 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202896 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202899 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202901 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202904 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.202913 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203694 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203702 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203705 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203709 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203712 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203715 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203718 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203720 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203723 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203726 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203729 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203732 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203734 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203737 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.205399 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203740 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203743 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203745 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203749 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203751 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203754 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203758 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203762 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203765 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203768 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203771 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203773 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203777 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203781 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203784 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203786 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203789 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203792 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203802 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.205949 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203805 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203808 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203810 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203813 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203815 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203818 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203820 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203824 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203828 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203830 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203833 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203836 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203838 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203841 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203843 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203846 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203849 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203852 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203854 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203857 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.206472 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203859 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203862 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203864 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203867 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203869 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203872 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203874 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203877 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203879 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203882 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203884 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203887 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203891 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203893 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203896 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203899 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203901 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203904 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203906 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203909 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.207022 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203912 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203914 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203917 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203919 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203922 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203925 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203928 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203930 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203933 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203935 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203938 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203940 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203943 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203946 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203949 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203951 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203953 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203956 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203959 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.207546 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203961 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203964 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203967 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203970 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203972 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203975 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203979 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203982 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203984 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203987 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203990 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203992 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203995 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.203997 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204499 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204505 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204508 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204512 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204517 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.208059 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204520 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204523 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204526 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204529 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204532 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204535 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204537 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204540 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204542 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204545 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204548 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204550 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204553 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204555 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204558 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204560 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204563 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204580 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204586 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204589 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.208614 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204592 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204594 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204597 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204600 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204602 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204605 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204607 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204610 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204613 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204615 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204618 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204621 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204623 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204626 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204629 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204631 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204633 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204636 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204638 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204641 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.209133 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204643 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204646 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204649 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204652 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204654 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204657 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204659 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204661 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204664 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204667 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204671 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204674 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204678 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204680 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204683 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204686 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204688 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204691 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204693 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.209667 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204696 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204698 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204701 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204704 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204706 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204709 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204711 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204714 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204716 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204719 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204721 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204724 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204726 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204729 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204731 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204734 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204737 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204739 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204742 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204745 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.210179 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204747 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204750 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204753 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204757 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204760 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204762 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204767 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204771 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204774 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204777 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204780 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204783 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204785 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204787 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204790 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204792 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204795 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204798 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204800 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.210733 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204803 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204805 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.204808 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204904 4980 flags.go:64] FLAG: --address="0.0.0.0" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204911 4980 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204917 4980 flags.go:64] FLAG: --anonymous-auth="true" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204922 4980 flags.go:64] FLAG: --application-metrics-count-limit="100" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204926 4980 flags.go:64] FLAG: --authentication-token-webhook="false" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204930 4980 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204934 4980 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204938 4980 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204941 4980 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204944 4980 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204948 4980 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204953 4980 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204957 4980 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204961 4980 flags.go:64] FLAG: --cgroup-root="" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204964 4980 flags.go:64] FLAG: --cgroups-per-qos="true" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204967 4980 flags.go:64] FLAG: --client-ca-file="" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204970 4980 flags.go:64] FLAG: --cloud-provider="external" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204974 4980 flags.go:64] FLAG: --cluster-dns="[]" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204979 4980 flags.go:64] FLAG: --cluster-domain="" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204982 4980 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Nov 03 17:33:40.211226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204985 4980 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204988 4980 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204991 4980 flags.go:64] FLAG: --container-log-max-files="5" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204995 4980 flags.go:64] FLAG: --container-log-max-size="10Mi" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.204998 4980 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205002 4980 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205005 4980 flags.go:64] FLAG: --containerd-namespace="k8s.io" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205008 4980 flags.go:64] FLAG: --contention-profiling="false" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205012 4980 flags.go:64] FLAG: --cpu-cfs-quota="true" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205015 4980 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205018 4980 flags.go:64] FLAG: --cpu-manager-policy="none" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205021 4980 flags.go:64] FLAG: --cpu-manager-policy-options="" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205025 4980 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205028 4980 flags.go:64] FLAG: --enable-controller-attach-detach="true" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205031 4980 flags.go:64] FLAG: --enable-debugging-handlers="true" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205034 4980 flags.go:64] FLAG: --enable-load-reader="false" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205037 4980 flags.go:64] FLAG: --enable-server="true" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205040 4980 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205044 4980 flags.go:64] FLAG: --event-burst="100" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205047 4980 flags.go:64] FLAG: --event-qps="50" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205050 4980 flags.go:64] FLAG: --event-storage-age-limit="default=0" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205053 4980 flags.go:64] FLAG: --event-storage-event-limit="default=0" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205056 4980 flags.go:64] FLAG: --eviction-hard="" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205061 4980 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Nov 03 17:33:40.211866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205064 4980 flags.go:64] FLAG: --eviction-minimum-reclaim="" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205068 4980 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205071 4980 flags.go:64] FLAG: --eviction-soft="" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205074 4980 flags.go:64] FLAG: --eviction-soft-grace-period="" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205077 4980 flags.go:64] FLAG: --exit-on-lock-contention="false" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205080 4980 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205084 4980 flags.go:64] FLAG: --experimental-mounter-path="" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205087 4980 flags.go:64] FLAG: --fail-cgroupv1="true" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205090 4980 flags.go:64] FLAG: --fail-swap-on="true" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205093 4980 flags.go:64] FLAG: --feature-gates="" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205097 4980 flags.go:64] FLAG: --file-check-frequency="20s" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205100 4980 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205103 4980 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205106 4980 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205109 4980 flags.go:64] FLAG: --healthz-port="10248" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205112 4980 flags.go:64] FLAG: --help="false" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205115 4980 flags.go:64] FLAG: --hostname-override="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205118 4980 flags.go:64] FLAG: --housekeeping-interval="10s" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205121 4980 flags.go:64] FLAG: --http-check-frequency="20s" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205124 4980 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205137 4980 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205141 4980 flags.go:64] FLAG: --image-gc-high-threshold="85" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205144 4980 flags.go:64] FLAG: --image-gc-low-threshold="80" Nov 03 17:33:40.212531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205147 4980 flags.go:64] FLAG: --image-service-endpoint="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205150 4980 flags.go:64] FLAG: --kernel-memcg-notification="false" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205153 4980 flags.go:64] FLAG: --kube-api-burst="100" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205155 4980 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205158 4980 flags.go:64] FLAG: --kube-api-qps="50" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205161 4980 flags.go:64] FLAG: --kube-reserved="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205165 4980 flags.go:64] FLAG: --kube-reserved-cgroup="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205168 4980 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205172 4980 flags.go:64] FLAG: --kubelet-cgroups="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205175 4980 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205180 4980 flags.go:64] FLAG: --lock-file="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205183 4980 flags.go:64] FLAG: --log-cadvisor-usage="false" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205186 4980 flags.go:64] FLAG: --log-flush-frequency="5s" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205189 4980 flags.go:64] FLAG: --log-json-info-buffer-size="0" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205194 4980 flags.go:64] FLAG: --log-json-split-stream="false" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205197 4980 flags.go:64] FLAG: --log-text-info-buffer-size="0" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205200 4980 flags.go:64] FLAG: --log-text-split-stream="false" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205203 4980 flags.go:64] FLAG: --logging-format="text" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205206 4980 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205209 4980 flags.go:64] FLAG: --make-iptables-util-chains="true" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205212 4980 flags.go:64] FLAG: --manifest-url="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205215 4980 flags.go:64] FLAG: --manifest-url-header="" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205220 4980 flags.go:64] FLAG: --max-housekeeping-interval="15s" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205223 4980 flags.go:64] FLAG: --max-open-files="1000000" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205227 4980 flags.go:64] FLAG: --max-pods="110" Nov 03 17:33:40.213309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205230 4980 flags.go:64] FLAG: --maximum-dead-containers="-1" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205233 4980 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205236 4980 flags.go:64] FLAG: --memory-manager-policy="None" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205238 4980 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205241 4980 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205244 4980 flags.go:64] FLAG: --node-ip="0.0.0.0" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205247 4980 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205254 4980 flags.go:64] FLAG: --node-status-max-images="50" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205257 4980 flags.go:64] FLAG: --node-status-update-frequency="10s" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205260 4980 flags.go:64] FLAG: --oom-score-adj="-999" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205264 4980 flags.go:64] FLAG: --pod-cidr="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205266 4980 flags.go:64] FLAG: --pod-manifest-path="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205269 4980 flags.go:64] FLAG: --pod-max-pids="-1" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205272 4980 flags.go:64] FLAG: --pods-per-core="0" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205275 4980 flags.go:64] FLAG: --port="10250" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205281 4980 flags.go:64] FLAG: --protect-kernel-defaults="false" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205284 4980 flags.go:64] FLAG: --provider-id="aws:///us-west-1a/i-05a9d469a55b2235a" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205288 4980 flags.go:64] FLAG: --qos-reserved="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205291 4980 flags.go:64] FLAG: --read-only-port="10255" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205294 4980 flags.go:64] FLAG: --register-node="true" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205297 4980 flags.go:64] FLAG: --register-with-taints="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205301 4980 flags.go:64] FLAG: --registry-burst="10" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205304 4980 flags.go:64] FLAG: --registry-qps="5" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205306 4980 flags.go:64] FLAG: --reserved-cpus="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205310 4980 flags.go:64] FLAG: --reserved-memory="" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205314 4980 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Nov 03 17:33:40.214353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205317 4980 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205320 4980 flags.go:64] FLAG: --rotate-certificates="false" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205323 4980 flags.go:64] FLAG: --rotate-server-certificates="false" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205326 4980 flags.go:64] FLAG: --runonce="false" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205329 4980 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205332 4980 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205335 4980 flags.go:64] FLAG: --seccomp-default="false" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205339 4980 flags.go:64] FLAG: --serialize-image-pulls="true" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205341 4980 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205344 4980 flags.go:64] FLAG: --storage-driver-db="cadvisor" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205348 4980 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205350 4980 flags.go:64] FLAG: --storage-driver-password="root" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205353 4980 flags.go:64] FLAG: --storage-driver-secure="false" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205356 4980 flags.go:64] FLAG: --storage-driver-table="stats" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205359 4980 flags.go:64] FLAG: --storage-driver-user="root" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205362 4980 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205365 4980 flags.go:64] FLAG: --sync-frequency="1m0s" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205368 4980 flags.go:64] FLAG: --system-cgroups="" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205371 4980 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205376 4980 flags.go:64] FLAG: --system-reserved-cgroup="" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205379 4980 flags.go:64] FLAG: --tls-cert-file="" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205383 4980 flags.go:64] FLAG: --tls-cipher-suites="[]" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205387 4980 flags.go:64] FLAG: --tls-min-version="" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205392 4980 flags.go:64] FLAG: --tls-private-key-file="" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205395 4980 flags.go:64] FLAG: --topology-manager-policy="none" Nov 03 17:33:40.215310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205398 4980 flags.go:64] FLAG: --topology-manager-policy-options="" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205401 4980 flags.go:64] FLAG: --topology-manager-scope="container" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205404 4980 flags.go:64] FLAG: --v="2" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205409 4980 flags.go:64] FLAG: --version="false" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205413 4980 flags.go:64] FLAG: --vmodule="" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205417 4980 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205421 4980 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205548 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205551 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205554 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205556 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205559 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205562 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205564 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205580 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205583 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205585 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205588 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205590 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205593 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205596 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.216009 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205599 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205602 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205604 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205607 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205610 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205613 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205615 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205619 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205622 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205627 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205630 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205633 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205636 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205639 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205642 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205644 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205647 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205649 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205652 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.216548 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205655 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205658 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205660 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205663 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205665 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205668 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205670 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205673 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205675 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205678 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205680 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205683 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205685 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205688 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205690 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205693 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205695 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205698 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205701 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.217241 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205705 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205709 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205712 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205717 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205719 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205722 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205725 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205728 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205730 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205733 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205735 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205738 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205741 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205744 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205747 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205749 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205752 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205755 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205758 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.217935 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205760 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205763 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205765 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205768 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205771 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205773 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205775 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205778 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205781 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205783 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205786 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205788 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205791 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205794 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205797 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205800 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205804 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205806 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205809 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.218523 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205811 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205814 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205817 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205819 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205822 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205824 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205827 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205830 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205832 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205835 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205838 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205840 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205843 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205845 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205848 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205850 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.219066 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.205875 4980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205929 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205934 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205937 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205939 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205942 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205944 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205947 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205950 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205953 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205955 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205960 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205962 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205966 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205970 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205973 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205975 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205978 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205980 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205983 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.219559 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205985 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205988 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205991 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205993 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205996 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.205999 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206001 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206004 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206006 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206009 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206012 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206014 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206017 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206019 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206022 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206025 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206028 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206032 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206034 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.220260 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206038 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206040 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206043 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206045 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206049 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206052 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206054 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206058 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206061 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206064 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206066 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206069 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206071 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206074 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206076 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206079 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206081 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206084 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206086 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206089 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.220859 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206092 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206095 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206097 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206100 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206102 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206105 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206109 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206112 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206115 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206117 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206120 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206122 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206125 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206128 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206131 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206134 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206140 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206143 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206146 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.221699 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206150 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206152 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206155 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206158 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206160 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206163 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206166 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206168 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206171 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206173 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206176 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206178 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206181 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206184 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206187 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206189 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206192 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206194 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206197 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206200 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.222538 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206202 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206205 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206207 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206210 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206212 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206215 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206218 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206220 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.206224 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.210310 4980 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.210326 4980 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.210344 4980 watchdog_linux.go:95] "Systemd watchdog is not enabled" Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.210349 4980 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210406 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210411 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210415 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210418 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210422 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210427 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.223439 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210430 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210433 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210436 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210438 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210441 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210444 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210447 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210449 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210452 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210455 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210457 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210460 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210463 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210474 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210477 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210481 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210483 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210486 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210489 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210492 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.224329 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210494 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210497 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210500 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210511 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210514 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210517 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210520 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210523 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210525 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210528 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210531 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210534 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210536 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210539 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210541 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210544 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210547 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210549 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210552 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210555 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.225203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210558 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210560 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210563 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210565 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210585 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210588 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210591 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210593 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210598 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210601 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210605 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210607 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210610 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210613 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210616 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210618 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210627 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210630 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210632 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210635 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.226146 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210638 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210640 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210643 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210645 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210648 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210650 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210653 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210655 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210658 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210661 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210663 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210666 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210668 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210670 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210673 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210675 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210679 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210681 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210683 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.227056 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210686 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210689 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210691 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210694 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210696 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210699 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210702 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210704 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210707 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210709 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210717 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210720 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210723 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210725 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210728 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210730 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210733 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210735 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210744 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210747 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.227933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210750 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.210779 4980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210976 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210982 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210986 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210988 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210991 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210994 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210997 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.210999 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211002 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211005 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211007 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211010 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211012 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.228904 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211015 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211018 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211020 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211023 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211025 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211028 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211030 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211033 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211043 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211045 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211048 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211050 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211053 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211056 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211058 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211061 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211063 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211065 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211068 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211071 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.229651 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211073 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211076 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211078 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211081 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211083 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211086 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211088 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211091 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211093 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211096 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211098 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211101 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211103 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211106 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211108 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211111 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211113 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211116 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211119 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211121 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.230756 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211124 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211132 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211134 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211137 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211140 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211142 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211144 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211147 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211149 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211152 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211154 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211157 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211159 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211162 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211165 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211168 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211170 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211173 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211175 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.231712 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211178 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211180 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211183 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211185 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211188 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211191 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211193 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211195 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211199 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211203 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211206 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211209 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211211 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211214 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211217 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211225 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211228 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211231 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211233 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.232621 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211236 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211239 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211241 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211244 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211247 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211250 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211252 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211255 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211258 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211260 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211263 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211265 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211268 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211270 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211273 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.211295 4980 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Nov 03 17:33:40.233517 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211332 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211337 4980 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211340 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211344 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211347 4980 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211351 4980 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211354 4980 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211357 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211360 4980 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211363 4980 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211365 4980 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211368 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211371 4980 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211379 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211382 4980 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211384 4980 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211387 4980 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211389 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Nov 03 17:33:40.234433 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211392 4980 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211394 4980 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211397 4980 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211399 4980 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211402 4980 feature_gate.go:386] unrecognized feature gate: NewOLM Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211405 4980 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211407 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211409 4980 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211412 4980 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211414 4980 feature_gate.go:386] unrecognized feature gate: Example Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211417 4980 feature_gate.go:386] unrecognized feature gate: EVPN Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211420 4980 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211422 4980 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211425 4980 feature_gate.go:386] unrecognized feature gate: EventTTL Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211427 4980 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211430 4980 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211432 4980 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211435 4980 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211438 4980 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211440 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Nov 03 17:33:40.235324 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211442 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211445 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211448 4980 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211450 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211453 4980 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211455 4980 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211458 4980 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211460 4980 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211468 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211471 4980 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211474 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211479 4980 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211482 4980 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211485 4980 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211487 4980 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211490 4980 feature_gate.go:386] unrecognized feature gate: InsightsConfig Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211493 4980 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211502 4980 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211504 4980 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211507 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Nov 03 17:33:40.236235 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211509 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211512 4980 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211514 4980 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211517 4980 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211519 4980 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211522 4980 feature_gate.go:386] unrecognized feature gate: NetworkConnect Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211524 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211527 4980 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211529 4980 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211532 4980 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211534 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211537 4980 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211539 4980 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211542 4980 feature_gate.go:386] unrecognized feature gate: TLSAdherence Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211544 4980 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211547 4980 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211549 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211552 4980 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211554 4980 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Nov 03 17:33:40.237163 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211556 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211559 4980 feature_gate.go:386] unrecognized feature gate: OVNObservability Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211590 4980 feature_gate.go:386] unrecognized feature gate: Example2 Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211595 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211599 4980 feature_gate.go:386] unrecognized feature gate: KMSEncryption Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211602 4980 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211606 4980 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211609 4980 feature_gate.go:386] unrecognized feature gate: OSStreams Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211613 4980 feature_gate.go:386] unrecognized feature gate: DualReplica Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211616 4980 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211618 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211621 4980 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211624 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211626 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211629 4980 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211632 4980 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211634 4980 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211637 4980 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211640 4980 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211642 4980 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Nov 03 17:33:40.238094 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211645 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211647 4980 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211650 4980 feature_gate.go:386] unrecognized feature gate: SignatureStores Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211653 4980 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211657 4980 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211659 4980 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211663 4980 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211665 4980 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:40.211668 4980 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.211727 4980 server.go:957] "Client rotation is on, will bootstrap in background" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.212751 4980 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.212805 4980 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.213176 4980 server.go:1014] "Starting client certificate rotation" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.214542 4980 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.215049 4980 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-02 17:27:59 +0000 UTC" deadline="2028-08-17 03:19:33.815107078 +0000 UTC" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.215125 4980 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15657h45m53.599985895s" Nov 03 17:33:40.240643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.215229 4980 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:33:40.241279 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.216301 4980 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Nov 03 17:33:40.241279 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.217955 4980 log.go:25] "Validated CRI v1 runtime API" Nov 03 17:33:40.241279 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.219734 4980 log.go:25] "Validated CRI v1 image API" Nov 03 17:33:40.241279 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.220541 4980 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Nov 03 17:33:40.241279 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.223277 4980 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b02b216c-d8cf-4dd6-b4fc-9a69cab1708e:/dev/nvme0n1p4 d2f9b9a6-2918-4012-8830-47cd2ad8ac7f:/dev/nvme0n1p3] Nov 03 17:33:40.241593 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.223299 4980 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/02132f883a1d7004bbf2b5332eb6b5e468781411a4d1fcaa5a03c7961053c193/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/02132f883a1d7004bbf2b5332eb6b5e468781411a4d1fcaa5a03c7961053c193/userdata/shm major:0 minor:233 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3cf4b0281d310179605867818704d02af998c06f6a52e487b3cd4df1e73e13ea/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3cf4b0281d310179605867818704d02af998c06f6a52e487b3cd4df1e73e13ea/userdata/shm major:0 minor:237 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407/userdata/shm major:0 minor:60 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6f19982762c0d5067082948ddff301b09e190afa7498f2f71a2015571a776895/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6f19982762c0d5067082948ddff301b09e190afa7498f2f71a2015571a776895/userdata/shm major:0 minor:290 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06/userdata/shm major:0 minor:64 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865/userdata/shm major:0 minor:336 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd/userdata/shm major:0 minor:73 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04/userdata/shm major:0 minor:242 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e/userdata/shm major:0 minor:80 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188/userdata/shm major:0 minor:221 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/048b7350-b701-4e23-856e-67d3413dc370/volumes/kubernetes.io~projected/kube-api-access-zc79m:{mountpoint:/var/lib/kubelet/pods/048b7350-b701-4e23-856e-67d3413dc370/volumes/kubernetes.io~projected/kube-api-access-zc79m major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0767163e-8374-4f56-9d2f-e15e26fdcd2e/volumes/kubernetes.io~projected/kube-api-access-xpwnq:{mountpoint:/var/lib/kubelet/pods/0767163e-8374-4f56-9d2f-e15e26fdcd2e/volumes/kubernetes.io~projected/kube-api-access-xpwnq major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~projected/kube-api-access-fhgmr:{mountpoint:/var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~projected/kube-api-access-fhgmr major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~secret/serving-cert major:0 minor:216 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:47 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/tmp major:0 minor:50 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~projected/kube-api-access-kgzgp:{mountpoint:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~projected/kube-api-access-kgzgp major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/333b0df1-e4ba-4064-979c-655f7e52b9f0/volumes/kubernetes.io~projected/kube-api-access-9b22p:{mountpoint:/var/lib/kubelet/pods/333b0df1-e4ba-4064-979c-655f7e52b9f0/volumes/kubernetes.io~projected/kube-api-access-9b22p major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3ba272e4-ee5f-4abd-9c93-26f5a2563098/volumes/kubernetes.io~projected/kube-api-access-6dm74:{mountpoint:/var/lib/kubelet/pods/3ba272e4-ee5f-4abd-9c93-26f5a2563098/volumes/kubernetes.io~projected/kube-api-access-6dm74 major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~projected/kube-api-access-xddlm:{mountpoint:/var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~projected/kube-api-access-xddlm major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~secret/serving-cert major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5d987bf9-3048-4b49-ba64-010106066297/volumes/kubernetes.io~projected/kube-api-access-v88mt:{mountpoint:/var/lib/kubelet/pods/5d987bf9-3048-4b49-ba64-010106066297/volumes/kubernetes.io~projected/kube-api-access-v88mt major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:234 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/kube-api-access-ntgqq:{mountpoint:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/kube-api-access-ntgqq major:0 minor:231 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:211 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/62660df0-cd00-4e15-87ce-7d8177463419/volumes/kubernetes.io~projected/kube-api-access-h2hnz:{mountpoint:/var/lib/kubelet/pods/62660df0-cd00-4e15-87ce-7d8177463419/volumes/kubernetes.io~projected/kube-api-access-h2hnz major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/645841ea-aa0f-4baa-8923-a460dc301b5a/volumes/kubernetes.io~projected/kube-api-access-t8z6h:{mountpoint:/var/lib/kubelet/pods/645841ea-aa0f-4baa-8923-a460dc301b5a/volumes/kubernetes.io~projected/kube-api-access-t8z6h major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~projected/kube-api-access-jpb8d:{mountpoint:/var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~projected/kube-api-access-jpb8d major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~secret/cert major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~projected/kube-api-access-zjp2c:{mountpoint:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~projected/kube-api-access-zjp2c major:0 minor:227 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/default-certificate:{mountpoint:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/default-certificate major:0 minor:212 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/stats-auth:{mountpoint:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/stats-auth major:0 minor:213 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8aa2379e-1eed-4cce-a9df-e96665bf3fd9/volumes/kubernetes.io~projected/kube-api-access-g7682:{mountpoint:/var/lib/kubelet/pods/8aa2379e-1eed-4cce-a9df-e96665bf3fd9/volumes/kubernetes.io~projected/kube-api-access-g7682 major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8f4f29a5-0dfb-4515-9bd6-276c084934bb/volumes/kubernetes.io~projected/kube-api-access-695fn:{mountpoint:/var/lib/kubelet/pods/8f4f29a5-0dfb-4515-9bd6-276c084934bb/volumes/kubernetes.io~projected/kube-api-access-695fn major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92dc67cc-a063-43af-9a22-432e61ee10f4/volumes/kubernetes.io~projected/kube-api-access-q496r:{mountpoint:/var/lib/kubelet/pods/92dc67cc-a063-43af-9a22-432e61ee10f4/volumes/kubernetes.io~projected/kube-api-access-q496r major:0 minor:54 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/92e67f15-1027-4a0d-8491-736e2cd4bc04/volumes/kubernetes.io~projected/kube-api-access-d69bj:{mountpoint:/var/lib/kubelet/pods/92e67f15-1027-4a0d-8491-736e2cd4bc04/volumes/kubernetes.io~projected/kube-api-access-d69bj major:0 minor:223 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~projected/kube-api-access-72r2f:{mountpoint:/var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~projected/kube-api-access-72r2f major:0 minor:206 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~secret/serving-cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/bound-sa-token:{mountpoint:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/bound-sa-token major:0 minor:226 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/kube-api-access-r54kn:{mountpoint:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/kube-api-access-r54kn major:0 minor:229 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/image-registry-private-configuration:{mountpoint:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/image-registry-private-configuration major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/installation-pull-secrets:{mountpoint:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/installation-pull-secrets major:0 minor:214 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a7f26304-7fb0-49b7-8f59-d8dc02c88d09/volumes/kubernetes.io~projected/kube-api-access-txrr7:{mountpoint:/var/lib/kubelet/pods/a7f26304-7fb0-49b7-8f59-d8dc02c88d09/volumes/kubernetes.io~projected/kube-api-access-txrr7 major:0 minor:51 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~projected/kube-api-access-r2qk9:{mountpoint:/var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~projected/kube-api-access-r2qk9 major:0 minor:230 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~secret/serving-cert:{mountpoint:/var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~secret/serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/bea67a96-85b1-41d3-b177-bfec1d287457/volumes/kubernetes.io~projected/kube-api-access-c8zdt:{mountpoint:/var/lib/kubelet/pods/bea67a96-85b1-41d3-b177-bfec1d287457/volumes/kubernetes.io~projected/kube-api-access-c8zdt major:0 minor:205 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4/volumes/kubernetes.io~secret/agent-certs major:0 minor:49 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb176382-bf16-4088-9f2e-0ec9932511f5/volumes/kubernetes.io~projected/kube-api-access-d499d:{mountpoint:/var/lib/kubelet/pods/cb176382-bf16-4088-9f2e-0ec9932511f5/volumes/kubernetes.io~projected/kube-api-access-d499d major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cbea6928-9579-4606-afd8-42fccd905154/volumes/kubernetes.io~projected/kube-api-access-gr8jh:{mountpoint:/var/lib/kubelet/pods/cbea6928-9579-4606-afd8-42fccd905154/volumes/kubernetes.io~projected/kube-api-access-gr8jh major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1796bb1-d79a-448a-9152-8570b92e43e6/volumes/kubernetes.io~projected/kube-api-access-7sbl5:{mountpoint:/var/lib/kubelet/pods/d1796bb1-d79a-448a-9152-8570b92e43e6/volumes/kubernetes.io~projected/kube-api-access-7sbl5 major:0 minor:52 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/defb7da5-8103-40ef-9607-49c56c281187/volumes/kubernetes.io~projected/kube-api-access-999ff:{mountpoint:/var/lib/kubelet/pods/defb7da5-8103-40ef-9607-49c56c281187/volumes/kubernetes.io~projected/kube-api-access-999ff major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dfad8e5c-4000-47b2-b39c-ea805fd6e315/volumes/kubernetes.io~projected/kube-api-access-hwc8l:{mountpoint:/var/lib/kubelet/pods/dfad8e5c-4000-47b2-b39c-ea805fd6e315/volumes/kubernetes.io~projected/kube-api-access-hwc8l major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~projected/kube-api-access-rnhsz:{mountpoint:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~projected/kube-api-access-rnhsz major:0 minor:53 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:48 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fedfb5ca-94b5-4850-8020-a7279aa49eae/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/fedfb5ca-94b5-4850-8020-a7279aa49eae/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:286 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/331f9ef950dd8502557b516baac50df8cf20ae8e3e10be46ab641d3c87eae7eb/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/f78dd5e81baf37552fa33b82dcaeeb9a7d0ebd7081c864d056ad6f05e0327d9f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/16bb14149d94db4abb94b140abf3d047c2a29d68f847728f123787fdbebf7c09/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/765b0bb604226f445fe2eefe3376c03d9ad0d5362cbc0975bdf8ec740e5e5d45/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/178ee57654407d5cd88c0de92e14dac0f6ada6c31f19424d8db71893f412abf7/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/a5adb59a5a3fcb1af7a3630d64b57cd6b7c1cea3871490f9f7f24518e4e64ef3/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/ccfd69a98325d7f1e5d6b83a51d369a3025015011292f038e35ba8e6aa8010fc/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/fccd7abe1f090213f74783ed680287c44da9f08f818dea1a518e5fbfbb45cf79/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/b2ebe73304635ebe99d3d4c1cdad9d79fbe5b8535b8b7fee97b0aa98f2dd2f44/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/3bf38c0becf3da4edb41b2fa04e6d13cd432b487b217cf2ce86c3b1b379b5764/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/5818558c708956b47f80e37b3ab8f8cf878e110773dc54636e8d1b61699843bf/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/4ad3d37adb1bec34c2346e4e08de844a729408bd474fb175e67fd8a9174e26bb/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-153:{mountpoint:/var/lib/containers/storage/overlay/ca53afd2ed88b5416ccc12e9d517e69defa0c6e3148bc28b7439167f3cabbb10/merged major:0 minor:153 fsType:overlay blockSize:0} overlay_0-155:{mountpoint:/var/lib/containers/storage/overlay/3e73251a17f2d92ece58c5ccbad9f561107267c92f9ff7ff65a0f6078b7fd2c3/merged major:0 minor:155 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/7b4a99ce8b8523a3d2fcafa0d130a8780c68b3624f6796b4b09e5718212e9ee8/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/fe8f8071899584b7be5a7fb997d49c06a56c9f4548aaf3196b41345288e2fa80/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/e0751fd0eca718ecc21d63c355e8e71ded61bc8c95f5a2009f0d44ee3e3b9e5a/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/99881d6fe06d8da11afec30464e7f9cdbb1b98c61d335f79e5ade5a7814f2def/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/de4a663568a67ddcab1f0429ef61ef1ede5e62327db33d9cc1968dc93c683888/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/a0beb812b9af8a924a8ec62325f333cc1a675bcbc2d1c1e3a9515cbd7a0af0b1/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/d3a7aa8015b244269db98aa5591cdcf637dc5a782056d17f0dfb0f634b4e7a75/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-248:{mountpoint:/var/lib/containers/storage/overlay/9280250b3a1bbdc3f2631f47c265e9dbd2caf7a0f6751682de2f1da46ec531ff/merged major:0 minor:248 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/b92840b88e356a50b9690fe87b8ab63ba0b5d7dc8851c6dee97fae309059d7fc/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/d96983c2da23bd0c2aa5308cf9dbb9ca322c22b48f1fc9d1059aa1a158dfb9d4/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/605954a816d0258b45f8f19ad762173c8d4cfa4224eb422a085e70b06410cc0d/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/d8b0d8037082b9d628ebcb2f3b8b5baabced6bc5fd91495d11db8338ab643bce/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-258:{mountpoint:/var/lib/containers/storage/overlay/8c791b9b268d41a51453030efdb4d8faefe908bb5746e707af0d714994d6bf2e/merged major:0 minor:258 fsType:overlay blockSize:0} overlay_0-260:{mountpoint:/var/lib/containers/storage/overlay/c027be5d3f070504a78c7333eee3bbd47fc90747ba39e32cca5162c864afb183/merged major:0 minor:260 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/cf2585eb04c2052c1665e9ba354d0ecee95355e8c6da508443398369fbb9a097/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/91ef2e99563f590f639960a2df64c9e2ea8cedfb1b3d0a4beacb515f0df3b296/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/979140f138a4916ff111ab053192349636d85368afb800853439f3cbbdcc1062/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/d7ad3e78e1610170c2a1844dcae84c6931cd9bc15cbe87e8be3d1e05886482ac/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-282:{mountpoint:/var/lib/containers/storage/overlay/b0b71440d5dc12d4eb954a3188ccfc4c84ca3c46d6c88f068afca0b5e5527467/merged major:0 minor:282 fsType:overlay blockSize:0} overlay_0-284:{mountpoint:/var/lib/containers/storage/overlay/1509cd582a6f8dc549ffbfb382b3609d5ee6852bacbd4b1e2997e61c1285cbd0/merged major:0 minor:284 fsType:overlay blockSize:0} overlay_0-292:{mountpoint:/var/lib/containers/storage/overlay/20fb273db0dbb3fb74a08374b72fa89ebe3e26be50492626409dd80abc7fb968/merged major:0 minor:292 fsType:overlay blockSize:0} overlay_0-294:{mountpoint:/var/lib/containers/storage/overlay/7fe934a0bc08b0b82dad3a35fe9d5dd364703a56ec95580eb51b778ef2731a9b/merged major:0 minor:294 fsType:overlay blockSize:0} overlay_0-296:{mountpoint:/var/lib/containers/storage/overlay/ea18d57d7f5384b0c8358634961f7e72e4bd4a56c2c8b6b03f476b5bad8aa5ee/merged major:0 minor:296 fsType:overlay blockSize:0} overlay_0-298:{mountpoint:/var/lib/containers/storage/overlay/3660e5d85a06122d9d2c6d44e475b95d1e263b4cf28a2fb3d2f43066edc3369c/merged major:0 minor:298 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/cf34cd9e87a029926e1d0cbe8106a49dd28c59c6dfd27a3dd0e77c9ae3899dd4/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/2b82e5109a10a28c29359e2c23ce5af5202687883e9007a68dc7e476b796ed71/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-313:{mountpoint:/var/lib/containers/storage/overlay/67cef18696d76b5183f7ff60a3a0e74ee1ded3130e8ad1df34912776af679483/merged major:0 minor:313 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/8d63e02ad00d4fa5c7bb4078345daed613e5fd1405e962f7916ea8cffcd24ebe/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-325:{mountpoint:/var/lib/containers/storage/overlay/71bbf7b8c5f90e4ed570ee822238a806608b7c51c21fe5660e1e8f7148ef2768/merged major:0 minor:325 fsType:overlay blockSize:0} overlay_0-342:{mountpoint:/var/lib/containers/storage/overlay/a63d1c11733d6644b10609a031544747d24bc950b0d9e37dd4662503f9033c4e/merged major:0 minor:342 fsType:overlay blockSize:0} overlay_0-344:{mountpoint:/var/lib/containers/storage/overlay/ee424ab946dc8e95dff5d66a1596e2e0cc6356aa225c13f63087d62a4fb1df9b/merged major:0 minor:344 fsType:overlay blockSize:0} overlay_0-346:{mountpoint:/var/lib/containers/storage/overlay/95ce9c91c95fbf240fb0a3fced094340d56fff0f066c0d2c5f5d1805587eaa99/merged major:0 minor:346 fsType:overlay blockSize:0} overlay_0-62:{mountpoint:/var/lib/containers/storage/overlay/e5cdffeaf56cfb52a6a9fa8e0903b2ae5920154fc77c48a1494109ad8df7b8c7/merged major:0 minor:62 fsType:overlay blockSize:0} overlay_0-66:{mountpoint:/var/lib/containers/storage/overlay/c32a95e399bf02f9ff42e2d5a8dc7c5034a6e570a1ed39fc2dc16acb0a1365ee/merged major:0 minor:66 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/35d14321a7c3e9066735d1c5500df6b3a0600614da60159af63d04596334cb3c/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-76:{mountpoint:/var/lib/containers/storage/overlay/03cc1450e0f5995f8d6df25a2956b95be114e8c8eb5a14912d38830b8237c32f/merged major:0 minor:76 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/2274e99f017aa5326b976ea6f242b45ec337409e5bd1009e4b64adb6135f9401/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/ff5f211a1440b2e23d2e10996c11074d179c91a76c6006879f24c087ab631ca3/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/004db91014faf926721425e74e786163ff72f2eaed9826b5891eda8c9aff5979/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/167e961f16462e5bd6a1337ea5fe4ab7f0aa5f5af136df3e2f1419875890be1e/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/99647dff94a480f20d25386c0da41020dee4c884f69f0091bf7ea93f4c2f45ad/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/1a8b50e206de2c66ecb8b975bdbc8c036d2f0e111a356a670afa41144f8e3503/merged major:0 minor:96 fsType:overlay blockSize:0}] Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.235686 4980 manager.go:217] Machine: {Timestamp:2026-11-03 17:33:40.234815546 +0000 UTC m=+0.063899162 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099671 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec26e167626d6e2779c16d1c346cc9bd SystemUUID:ec26e167-626d-6e27-79c1-6d1c346cc9bd BootID:d820c2d1-82d1-4db2-a53f-58ee86ac66e1 Filesystems:[{Device:/var/lib/kubelet/pods/92dc67cc-a063-43af-9a22-432e61ee10f4/volumes/kubernetes.io~projected/kube-api-access-q496r DeviceMajor:0 DeviceMinor:54 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:48 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5d987bf9-3048-4b49-ba64-010106066297/volumes/kubernetes.io~projected/kube-api-access-v88mt DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:226 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~projected/kube-api-access-fhgmr DeviceMajor:0 DeviceMinor:232 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-296 DeviceMajor:0 DeviceMinor:296 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f349a6f9b576934ce9e612edaad52320e8570161797e5221f180c38d42ed582e/userdata/shm DeviceMajor:0 DeviceMinor:80 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-153 DeviceMajor:0 DeviceMinor:153 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-325 DeviceMajor:0 DeviceMinor:325 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-346 DeviceMajor:0 DeviceMinor:346 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1796bb1-d79a-448a-9152-8570b92e43e6/volumes/kubernetes.io~projected/kube-api-access-7sbl5 DeviceMajor:0 DeviceMinor:52 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ae5aaf22333d14e23e52940a3ba34bba36eb05ae2736a1e8fc4eb6d82edd47cd/userdata/shm DeviceMajor:0 DeviceMinor:73 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8aa2379e-1eed-4cce-a9df-e96665bf3fd9/volumes/kubernetes.io~projected/kube-api-access-g7682 DeviceMajor:0 DeviceMinor:241 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-76 DeviceMajor:0 DeviceMinor:76 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55f734c728ed2fb248436d6b48dc7f0a9f7cf62dd91a139036d6e895d61d04da/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/92e67f15-1027-4a0d-8491-736e2cd4bc04/volumes/kubernetes.io~projected/kube-api-access-d69bj DeviceMajor:0 DeviceMinor:223 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-292 DeviceMajor:0 DeviceMinor:292 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volumes/kubernetes.io~projected/kube-api-access-rnhsz DeviceMajor:0 DeviceMinor:53 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fcb7ff97131ffdbbeea68d0c42d287262678d2c6f53074f18757d55bbcc2b188/userdata/shm DeviceMajor:0 DeviceMinor:221 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-260 DeviceMajor:0 DeviceMinor:260 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fedfb5ca-94b5-4850-8020-a7279aa49eae/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:286 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-155 DeviceMajor:0 DeviceMinor:155 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-298 DeviceMajor:0 DeviceMinor:298 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7d1919fb428ad09787ef1e4e40ea856f2e47294b0a9fff1f2c18fd77cc1bd718/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3e6e80aa85a3f7e8c6843609c7aae2d1ce9d4ce8f5afd8d1acb82e02ca051f89/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0a59ced0-24a7-45f0-a0c7-de168cbc59a7/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:216 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:50 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/17bd6ec4e0e299eef946a5691e59addaad7df159ad8daf47c678fa21db31497f/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/048b7350-b701-4e23-856e-67d3413dc370/volumes/kubernetes.io~projected/kube-api-access-zc79m DeviceMajor:0 DeviceMinor:312 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/883b8f4814041fd8abd8ea5926c6931f58cec943b4039fbeea7c02dff9740865/userdata/shm DeviceMajor:0 DeviceMinor:336 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cbea6928-9579-4606-afd8-42fccd905154/volumes/kubernetes.io~projected/kube-api-access-gr8jh DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-313 DeviceMajor:0 DeviceMinor:313 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/02132f883a1d7004bbf2b5332eb6b5e468781411a4d1fcaa5a03c7961053c193/userdata/shm DeviceMajor:0 DeviceMinor:233 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:236 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~projected/kube-api-access-r54kn DeviceMajor:0 DeviceMinor:229 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-248 DeviceMajor:0 DeviceMinor:248 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/62660df0-cd00-4e15-87ce-7d8177463419/volumes/kubernetes.io~projected/kube-api-access-h2hnz DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fe747a0c-5a1d-4db3-876a-f26e1852450e/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/default-certificate DeviceMajor:0 DeviceMinor:212 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/installation-pull-secrets DeviceMajor:0 DeviceMinor:214 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/bound-sa-token DeviceMajor:0 DeviceMinor:234 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~projected/kube-api-access-kgzgp DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0a4481e54fbb1cd9a4e39263b7e7b074048825fb16261cc145bab4e4709ce75a/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~projected/kube-api-access-zjp2c DeviceMajor:0 DeviceMinor:227 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/defb7da5-8103-40ef-9607-49c56c281187/volumes/kubernetes.io~projected/kube-api-access-999ff DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/67333be4-e8fc-47bd-89f3-5873637f8193/volumes/kubernetes.io~projected/kube-api-access-jpb8d DeviceMajor:0 DeviceMinor:239 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-66 DeviceMajor:0 DeviceMinor:66 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/788e811313999e7ef5b687769a40af7c17d8fe708e8219ae3f67a4e1d512725c/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:211 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-284 DeviceMajor:0 DeviceMinor:284 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6f19982762c0d5067082948ddff301b09e190afa7498f2f71a2015571a776895/userdata/shm DeviceMajor:0 DeviceMinor:290 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-294 DeviceMajor:0 DeviceMinor:294 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-342 DeviceMajor:0 DeviceMinor:342 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8f4f29a5-0dfb-4515-9bd6-276c084934bb/volumes/kubernetes.io~projected/kube-api-access-695fn DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-62 DeviceMajor:0 DeviceMinor:62 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c987e5cfe31de0f04eeecc4bbb3444967698c25bbe8cd0917680aebc3f3a1705/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~secret/serving-cert DeviceMajor:0 DeviceMinor:202 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-282 DeviceMajor:0 DeviceMinor:282 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0767163e-8374-4f56-9d2f-e15e26fdcd2e/volumes/kubernetes.io~projected/kube-api-access-xpwnq DeviceMajor:0 DeviceMinor:240 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-258 DeviceMajor:0 DeviceMinor:258 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/524f53ac3e56af52a7c6904c373db54662cb4244f16311ba187259628e20f9fc/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407/userdata/shm DeviceMajor:0 DeviceMinor:60 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9584d3a2-671c-48e3-89f9-dfd8e38db4ab/volumes/kubernetes.io~projected/kube-api-access-72r2f DeviceMajor:0 DeviceMinor:206 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/86cd6e5e387b417fa465888d29c1375fe1f4ee88d03e1ff28624ee9abe06b26c/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/cba914401a591b2c9481bafd9cdd8e8c84ed92f9923dff437af754ee4f37d140/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/333b0df1-e4ba-4064-979c-655f7e52b9f0/volumes/kubernetes.io~projected/kube-api-access-9b22p DeviceMajor:0 DeviceMinor:287 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/var/lib/kubelet/pods/32a05a82-dc96-44b1-b348-bef87dcc963c/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:47 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:49 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06/userdata/shm DeviceMajor:0 DeviceMinor:64 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/94312bef90f0d7584a94e4bb9adefcaf57d4342cb66e1e742a9b67390f01b6bc/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes/kubernetes.io~secret/image-registry-private-configuration DeviceMajor:0 DeviceMinor:218 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7e7acc31b994aa3fa98f49afca84660c36d0ae7d7b27483bcd8a21c36e8857cf/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes/kubernetes.io~projected/kube-api-access-ntgqq DeviceMajor:0 DeviceMinor:231 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cb176382-bf16-4088-9f2e-0ec9932511f5/volumes/kubernetes.io~projected/kube-api-access-d499d DeviceMajor:0 DeviceMinor:327 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-344 DeviceMajor:0 DeviceMinor:344 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/645841ea-aa0f-4baa-8923-a460dc301b5a/volumes/kubernetes.io~projected/kube-api-access-t8z6h DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dfad8e5c-4000-47b2-b39c-ea805fd6e315/volumes/kubernetes.io~projected/kube-api-access-hwc8l DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/bea67a96-85b1-41d3-b177-bfec1d287457/volumes/kubernetes.io~projected/kube-api-access-c8zdt DeviceMajor:0 DeviceMinor:205 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3cf4b0281d310179605867818704d02af998c06f6a52e487b3cd4df1e73e13ea/userdata/shm DeviceMajor:0 DeviceMinor:237 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a7f26304-7fb0-49b7-8f59-d8dc02c88d09/volumes/kubernetes.io~projected/kube-api-access-txrr7 DeviceMajor:0 DeviceMinor:51 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/56cf9464-6022-41a2-b0bc-be87d6b3b90e/volumes/kubernetes.io~projected/kube-api-access-xddlm DeviceMajor:0 DeviceMinor:207 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/3ba272e4-ee5f-4abd-9c93-26f5a2563098/volumes/kubernetes.io~projected/kube-api-access-6dm74 DeviceMajor:0 DeviceMinor:224 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b165161d322316ccc543ccd7c7a0b3eb73a391e3effbd67bd1f44ad59cd92f04/userdata/shm DeviceMajor:0 DeviceMinor:242 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8d340387503ebd1619021a9f7eb8420c98210b71bb21d65217fd19b563d46ad0/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6a6cf569-649e-420f-8fcf-3c1443782d9e/volumes/kubernetes.io~secret/stats-auth DeviceMajor:0 DeviceMinor:213 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/volumes/kubernetes.io~projected/kube-api-access-r2qk9 DeviceMajor:0 DeviceMinor:230 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:02132f883a1d700 MacAddress:c2:03:9e:c6:fb:30 Speed:10000 Mtu:8901} {Name:0a4481e54fbb1cd MacAddress:6e:df:89:12:30:00 Speed:10000 Mtu:8901} {Name:17bd6ec4e0e299e MacAddress:8a:6c:fb:6d:b5:dd Speed:10000 Mtu:8901} {Name:3cf4b0281d31017 MacAddress:5e:bb:9e:a4:de:bc Speed:10000 Mtu:8901} {Name:3e6e80aa85a3f7e MacAddress:b6:f7:ff:ee:b4:ca Speed:10000 Mtu:8901} {Name:524f53ac3e56af5 MacAddress:96:14:a4:48:a9:8a Speed:10000 Mtu:8901} {Name:6f19982762c0d50 MacAddress:32:5b:c9:cf:f4:b1 Speed:10000 Mtu:8901} {Name:7e7acc31b994aa3 MacAddress:42:e2:74:73:41:30 Speed:10000 Mtu:8901} {Name:86cd6e5e387b417 MacAddress:92:f8:a8:4a:40:61 Speed:10000 Mtu:8901} {Name:883b8f4814041fd MacAddress:22:87:bc:28:45:0f Speed:10000 Mtu:8901} {Name:b165161d322316c MacAddress:aa:84:c5:52:d8:d2 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:0f:d8:52:7a:4b Speed:0 Mtu:9001} {Name:br-int MacAddress:8a:9e:ff:26:ff:77 Speed:0 Mtu:8901} {Name:cba914401a591b2 MacAddress:ae:9b:ec:30:9a:84 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:0f:d8:52:7a:4b Speed:0 Mtu:9001} {Name:fcb7ff97131ffdb MacAddress:2a:2e:99:60:cd:0e Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:a2:e0:30:5e:27:b5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:5e:c6:6a:1f:ec:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.236300 4980 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.236413 4980 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.236874 4980 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.236903 4980 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-234.us-west-1.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237116 4980 topology_manager.go:143] "Creating topology manager with none policy" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237128 4980 container_manager_linux.go:311] "Creating device plugin manager" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237144 4980 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237167 4980 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237260 4980 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237487 4980 state_mem.go:41] "Initialized" logger="CPUManager state memory" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237705 4980 server.go:1235] "Using root directory" path="/var/lib/kubelet" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237757 4980 kubelet.go:493] "Attempting to sync node with API server" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237776 4980 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237791 4980 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237803 4980 kubelet.go:405] "Adding apiserver pod source" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.237813 4980 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.239916 4980 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.239948 4980 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:33:40.241921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.240935 4980 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Nov 03 17:33:40.243370 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243353 4980 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Nov 03 17:33:40.243370 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243367 4980 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Nov 03 17:33:40.243478 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243460 4980 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Nov 03 17:33:40.243754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243738 4980 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Nov 03 17:33:40.243834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243770 4980 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Nov 03 17:33:40.243887 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243861 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Nov 03 17:33:40.243887 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243874 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Nov 03 17:33:40.243887 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243881 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Nov 03 17:33:40.243887 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243888 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243894 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243901 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243908 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243915 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243923 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243930 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243940 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243950 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243975 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Nov 03 17:33:40.244061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.243982 4980 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Nov 03 17:33:40.247521 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.247506 4980 server.go:1263] "Started kubelet" Nov 03 17:33:40.247971 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.247579 4980 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Nov 03 17:33:40.247971 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.247676 4980 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Nov 03 17:33:40.247971 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.247757 4980 server_v1.go:49] "podresources" method="list" useActivePods=true Nov 03 17:33:40.248160 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.248053 4980 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Nov 03 17:33:40.248138 ip-10-0-139-234 systemd[1]: Started Kubernetes Kubelet. Nov 03 17:33:40.249303 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.249281 4980 server.go:317] "Adding debug handlers to kubelet server" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253374 4980 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253534 4980 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.253549 4980 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253623 4980 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-02 17:28:00 +0000 UTC" deadline="2028-05-17 03:20:54.083442331 +0000 UTC" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253668 4980 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13449h47m13.829777366s" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253709 4980 volume_manager.go:309] "The desired_state_of_world populator starts" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253718 4980 volume_manager.go:311] "Starting Kubelet Volume Manager" Nov 03 17:33:40.254439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.253829 4980 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.256772 4980 factory.go:55] Registering systemd factory Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.256795 4980 factory.go:223] Registration of the systemd container factory successfully Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257086 4980 factory.go:153] Registering CRI-O factory Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257098 4980 factory.go:223] Registration of the crio container factory successfully Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257172 4980 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257195 4980 factory.go:103] Registering Raw factory Nov 03 17:33:40.257494 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257206 4980 manager.go:1199] Started watching for new ooms in manager Nov 03 17:33:40.257888 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.257539 4980 manager.go:319] Starting recovery of all containers Nov 03 17:33:40.258404 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258362 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad57e2f-34f9-4ad6-82d7-3ad939d0fef4" volumeName="kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258405 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258418 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a05a82-dc96-44b1-b348-bef87dcc963c" volumeName="kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258430 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a05a82-dc96-44b1-b348-bef87dcc963c" volumeName="kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258443 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32a05a82-dc96-44b1-b348-bef87dcc963c" volumeName="kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258455 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258475 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4f29a5-0dfb-4515-9bd6-276c084934bb" volumeName="kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn" seLinuxMountContext="" Nov 03 17:33:40.258489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258486 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1796bb1-d79a-448a-9152-8570b92e43e6" volumeName="kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258500 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="048b7350-b701-4e23-856e-67d3413dc370" volumeName="kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258513 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0767163e-8374-4f56-9d2f-e15e26fdcd2e" volumeName="kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258527 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5d987bf9-3048-4b49-ba64-010106066297" volumeName="kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258539 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6cf569-649e-420f-8fcf-3c1443782d9e" volumeName="kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258551 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258593 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="defb7da5-8103-40ef-9607-49c56c281187" volumeName="kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258608 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a59ced0-24a7-45f0-a0c7-de168cbc59a7" volumeName="kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258617 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ba272e4-ee5f-4abd-9c93-26f5a2563098" volumeName="kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258628 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258640 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dc67cc-a063-43af-9a22-432e61ee10f4" volumeName="kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258649 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258660 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258671 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62660df0-cd00-4e15-87ce-7d8177463419" volumeName="kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258682 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258698 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258710 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258722 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258734 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67333be4-e8fc-47bd-89f3-5873637f8193" volumeName="kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258745 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92e67f15-1027-4a0d-8491-736e2cd4bc04" volumeName="kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258756 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cbea6928-9579-4606-afd8-42fccd905154" volumeName="kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258788 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="048b7350-b701-4e23-856e-67d3413dc370" volumeName="kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258803 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0767163e-8374-4f56-9d2f-e15e26fdcd2e" volumeName="kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258816 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6cf569-649e-420f-8fcf-3c1443782d9e" volumeName="kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258828 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258840 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a59ced0-24a7-45f0-a0c7-de168cbc59a7" volumeName="kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258851 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="62660df0-cd00-4e15-87ce-7d8177463419" volumeName="kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258863 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f26304-7fb0-49b7-8f59-d8dc02c88d09" volumeName="kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258875 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe747a0c-5a1d-4db3-876a-f26e1852450e" volumeName="kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258887 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="333b0df1-e4ba-4064-979c-655f7e52b9f0" volumeName="kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258901 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3ba272e4-ee5f-4abd-9c93-26f5a2563098" volumeName="kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258913 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="645841ea-aa0f-4baa-8923-a460dc301b5a" volumeName="kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258924 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn" seLinuxMountContext="" Nov 03 17:33:40.258928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258938 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe747a0c-5a1d-4db3-876a-f26e1852450e" volumeName="kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258951 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fedfb5ca-94b5-4850-8020-a7279aa49eae" volumeName="kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258964 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="645841ea-aa0f-4baa-8923-a460dc301b5a" volumeName="kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258977 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.258989 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67333be4-e8fc-47bd-89f3-5873637f8193" volumeName="kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259000 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8aa2379e-1eed-4cce-a9df-e96665bf3fd9" volumeName="kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259014 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9584d3a2-671c-48e3-89f9-dfd8e38db4ab" volumeName="kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259026 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f26304-7fb0-49b7-8f59-d8dc02c88d09" volumeName="kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259039 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" volumeName="kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259054 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="bea67a96-85b1-41d3-b177-bfec1d287457" volumeName="kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259066 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cad57e2f-34f9-4ad6-82d7-3ad939d0fef4" volumeName="kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259078 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9584d3a2-671c-48e3-89f9-dfd8e38db4ab" volumeName="kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259091 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9584d3a2-671c-48e3-89f9-dfd8e38db4ab" volumeName="kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259103 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a7f26304-7fb0-49b7-8f59-d8dc02c88d09" volumeName="kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259116 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe747a0c-5a1d-4db3-876a-f26e1852450e" volumeName="kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259128 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259148 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259160 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" volumeName="kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259190 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb176382-bf16-4088-9f2e-0ec9932511f5" volumeName="kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259203 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0a59ced0-24a7-45f0-a0c7-de168cbc59a7" volumeName="kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259215 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dc67cc-a063-43af-9a22-432e61ee10f4" volumeName="kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259227 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dc67cc-a063-43af-9a22-432e61ee10f4" volumeName="kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259239 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92e67f15-1027-4a0d-8491-736e2cd4bc04" volumeName="kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259253 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe747a0c-5a1d-4db3-876a-f26e1852450e" volumeName="kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259268 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259281 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259293 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" volumeName="kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259305 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6a6cf569-649e-420f-8fcf-3c1443782d9e" volumeName="kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259318 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8f4f29a5-0dfb-4515-9bd6-276c084934bb" volumeName="kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259329 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="92dc67cc-a063-43af-9a22-432e61ee10f4" volumeName="kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259354 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" volumeName="kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259367 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dfad8e5c-4000-47b2-b39c-ea805fd6e315" volumeName="kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259380 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0767163e-8374-4f56-9d2f-e15e26fdcd2e" volumeName="kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259402 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" volumeName="kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259415 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fe747a0c-5a1d-4db3-876a-f26e1852450e" volumeName="kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259426 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2be7ebc7-1f75-4f7b-a18e-7d8103188551" volumeName="kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259438 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="56cf9464-6022-41a2-b0bc-be87d6b3b90e" volumeName="kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259450 4980 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" volumeName="kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token" seLinuxMountContext="" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259473 4980 reconstruct.go:97] "Volume reconstruction finished" Nov 03 17:33:40.260324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.259480 4980 reconciler.go:29] "Reconciler: start to sync state" Nov 03 17:33:40.277345 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.277320 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:40.278812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.278792 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:40.284191 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.284140 4980 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Nov 03 17:33:40.284311 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.284291 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:40.285697 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.285674 4980 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Nov 03 17:33:40.285697 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.285698 4980 status_manager.go:249] "Starting to sync pod status with apiserver" Nov 03 17:33:40.285868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.285720 4980 kubelet.go:2521] "Starting kubelet main sync loop" Nov 03 17:33:40.285868 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.285757 4980 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Nov 03 17:33:40.288157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.288133 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:40.300012 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.299985 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:33:40.300427 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.300387 4980 generic.go:356] "Generic (PLEG): container finished" podID="fe747a0c-5a1d-4db3-876a-f26e1852450e" containerID="d9ef7366120b4c4fc283f770357906066214f7858437e278aa71d96becebba74" exitCode=1 Nov 03 17:33:40.317065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.317042 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:40.317157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.317078 4980 generic.go:356] "Generic (PLEG): container finished" podID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" exitCode=255 Nov 03 17:33:40.333609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333564 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="7fcf49cd20b833bc2385765e3d30f6195e9c633ab5cb46bdcbc9062cecf8881c" exitCode=0 Nov 03 17:33:40.333609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333609 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="676bd42e14226ec68974e117496da1f085043c2e2bdf3040c470dd146d432d55" exitCode=0 Nov 03 17:33:40.333725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333622 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="cd2832c39fae35844832a60f4199e45d792ad35aa717ffaf03736a4d27b33c4a" exitCode=0 Nov 03 17:33:40.333725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333632 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="3a1ed2b9fe770ae56f05d3cee5d5948170623b8c07a99808eb2303e25f9ea21e" exitCode=0 Nov 03 17:33:40.333725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333640 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="b562b8dec51e6b5d85b4156af6822cf37fff453cd8e04e8dd4bb0de03a310848" exitCode=0 Nov 03 17:33:40.333725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.333648 4980 generic.go:356] "Generic (PLEG): container finished" podID="92dc67cc-a063-43af-9a22-432e61ee10f4" containerID="5218a1651407dd57bbe0feff2cf21b6b43f9b718c4881383fba486832d2f5128" exitCode=0 Nov 03 17:33:40.337007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.336982 4980 generic.go:356] "Generic (PLEG): container finished" podID="4b39301f54ab728a5f22c5154db8ce29" containerID="931e553d221910359c664069ca9590512091222e8bf62c858db9e7f98d775e5f" exitCode=0 Nov 03 17:33:40.350289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.350269 4980 manager.go:324] Recovery completed Nov 03 17:33:40.353997 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.353963 4980 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:33:40.369372 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.369352 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.370426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.370368 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.370426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.370409 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.370426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.370421 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.372770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.372749 4980 cpu_manager.go:226] "Starting" policy="none" Nov 03 17:33:40.372770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.372764 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.372883 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.372765 4980 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Nov 03 17:33:40.372883 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.372866 4980 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Nov 03 17:33:40.373005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.372990 4980 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Nov 03 17:33:40.373042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373005 4980 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Nov 03 17:33:40.373042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373023 4980 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Nov 03 17:33:40.373042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373028 4980 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Nov 03 17:33:40.373042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373034 4980 policy_none.go:50] "Start" Nov 03 17:33:40.373147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373044 4980 memory_manager.go:187] "Starting memorymanager" policy="None" Nov 03 17:33:40.373147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373052 4980 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Nov 03 17:33:40.373551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373537 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.373631 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373593 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.373631 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.373611 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.374252 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.374236 4980 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Nov 03 17:33:40.374299 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.374254 4980 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Nov 03 17:33:40.374299 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.374262 4980 policy_none.go:44] "Start" Nov 03 17:33:40.378378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.378353 4980 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Nov 03 17:33:40.380914 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.380903 4980 manager.go:341] "Starting Device Plugin manager" Nov 03 17:33:40.380972 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.380926 4980 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Nov 03 17:33:40.380972 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.380934 4980 server.go:87] "Starting device plugin registration server" Nov 03 17:33:40.381179 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381167 4980 eviction_manager.go:194] "Eviction manager: starting control loop" Nov 03 17:33:40.381212 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381181 4980 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Nov 03 17:33:40.381293 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381277 4980 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Nov 03 17:33:40.381398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381362 4980 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Nov 03 17:33:40.381454 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381403 4980 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Nov 03 17:33:40.381454 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381410 4980 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Nov 03 17:33:40.381548 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.381463 4980 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-03T17:33:40.381392995Z","UUID":"56108ff4-b38b-4ab0-9029-7122163dd178","Handler":null,"Name":"","Endpoint":""} Nov 03 17:33:40.383175 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.383148 4980 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Nov 03 17:33:40.383259 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.383227 4980 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" Nov 03 17:33:40.383553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.383533 4980 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Nov 03 17:33:40.383671 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.383593 4980 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Nov 03 17:33:40.386111 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.386077 4980 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal"] Nov 03 17:33:40.386267 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.386206 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.387833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.387812 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.387906 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.387845 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.387906 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.387858 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.388210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.388194 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.388378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.388365 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.389007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.388990 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.389092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389012 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.389092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389021 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.389092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389026 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.389092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389049 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.389092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389063 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.389342 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.389163 4980 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.389491 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389461 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"32d0bc0f39e3274106efc0e449e45f7e","Type":"ContainerStarted","Data":"1055ebab309ff89dd078386a315b93a1d201a84963fa53c40a1d005c54021bcd"} Nov 03 17:33:40.389549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389512 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.389549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389528 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"32d0bc0f39e3274106efc0e449e45f7e","Type":"ContainerStarted","Data":"72e58f7c56ca4a1cac176d3a441cee8e3e52cba04e7cdd26811f8fd6f5620c06"} Nov 03 17:33:40.389663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389550 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerStarted","Data":"acb7bf89559a3afdadd8307716d2a8477e17d61b227d0710b7c332ac2c266052"} Nov 03 17:33:40.389663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389592 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerDied","Data":"931e553d221910359c664069ca9590512091222e8bf62c858db9e7f98d775e5f"} Nov 03 17:33:40.389663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.389612 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" event={"ID":"4b39301f54ab728a5f22c5154db8ce29","Type":"ContainerStarted","Data":"575ecaee875bff93e02cffa53a158ca55016d87f620c7c1591148992552b1407"} Nov 03 17:33:40.390196 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.390177 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.390277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.390206 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.390277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.390223 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.390372 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.390359 4980 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-234.us-west-1.compute.internal\" not found" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.450135 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.450114 4980 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Nov 03 17:33:40.450801 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.450752 4980 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:40.450801 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.450771 4980 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:40.452942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.452928 4980 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:40.454995 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.454981 4980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.482153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.482133 4980 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Nov 03 17:33:40.482991 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.482978 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientMemory" Nov 03 17:33:40.483049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.483007 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasNoDiskPressure" Nov 03 17:33:40.483049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.483020 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeHasSufficientPID" Nov 03 17:33:40.483118 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.483056 4980 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.502517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.502497 4980 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Nov 03 17:33:40.549865 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.549845 4980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:33:40.549935 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.549884 4980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.549935 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.549901 4980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.550967 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.550950 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.551037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.550983 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.551037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.551011 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.603269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.603241 4980 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.603381 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.603315 4980 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.603888 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.603872 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeNotReady" Nov 03 17:33:40.603951 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.603911 4980 setters.go:546] "Node became not ready" node="ip-10-0-139-234.us-west-1.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-03T17:33:40Z","lastTransitionTime":"2026-11-03T17:33:40Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"} Nov 03 17:33:40.617857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.617839 4980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:33:40.617908 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:40.617885 4980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.621264 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.621222 4980 nodeinfomanager.go:471] existing CSINode "ip-10-0-139-234.us-west-1.compute.internal" is owned by different node (oldNodeID="16b41938-02e8-4a8a-a5d3-c6b1782c83c1", newNodeID=""), cleaning up... Nov 03 17:33:40.651397 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651373 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.651488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651401 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.651488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651424 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.651488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651477 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/32d0bc0f39e3274106efc0e449e45f7e-config\") pod \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"32d0bc0f39e3274106efc0e449e45f7e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.651612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651486 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:40.651612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:40.651489 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4b39301f54ab728a5f22c5154db8ce29-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" (UID: \"4b39301f54ab728a5f22c5154db8ce29\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:41.238808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.238778 4980 apiserver.go:52] "Watching apiserver" Nov 03 17:33:41.345540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.345518 4980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:41.345686 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.345632 4980 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:41.394981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.394961 4980 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Nov 03 17:33:41.395825 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.395803 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-8jlfb","openshift-dns/node-resolver-gkxfp","kube-system/kas-connection-checker-685895c7c9-w5ndm","openshift-network-diagnostics/network-check-target-xp42v","openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj","openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm","openshift-image-registry/node-ca-gv7lt","openshift-network-operator/iptables-alerter-hp94c","openshift-ingress/router-default-5b97b666bc-r5q4p","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal","kube-system/konnectivity-agent-vwkww","kube-system/kas-connection-checker-685895c7c9-6tgbt","kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal","openshift-ovn-kubernetes/ovnkube-node-d8hsc","openshift-insights/insights-runtime-extractor-42zww","openshift-multus/multus-additional-cni-plugins-xmdk5","openshift-ingress-canary/ingress-canary-rtv8j","openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7","openshift-console-operator/console-operator-7cd56b8dbc-mpbwn","openshift-dns/dns-default-chggk","kube-system/global-pull-secret-syncer-fnq58","openshift-image-registry/image-registry-6b4c9976fc-ckgdt","openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln","openshift-multus/network-metrics-daemon-tlq8b","openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc","openshift-network-console/networking-console-plugin-86c478bb88-gcgd8","openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8","openshift-multus/multus-knk2x","openshift-image-registry/image-registry-59985d89dd-5th7m","openshift-insights/insights-operator-7c4986d449-4x9kc","openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn","kube-system/kas-connection-checker-685895c7c9-p4zh5"] Nov 03 17:33:41.397175 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.397156 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.397508 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.397492 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.401264 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.401234 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.403339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.403320 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.403808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.403789 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:41.404123 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.404109 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:41.407065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.407048 4980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:33:41.407132 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.407090 4980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:41.408561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.408543 4980 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Nov 03 17:33:41.408650 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.408611 4980 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-234.us-west-1.compute.internal" Nov 03 17:33:41.410091 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.410077 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.410721 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.410709 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.411930 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.411918 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.451197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451130 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.451197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451146 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.451197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451148 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.451197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451132 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.451197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451181 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.451629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451275 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-config\"" Nov 03 17:33:41.451629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451308 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.451629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451425 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Nov 03 17:33:41.451629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.451440 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.452242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452202 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Nov 03 17:33:41.452242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452228 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Nov 03 17:33:41.452554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452294 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Nov 03 17:33:41.452554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452378 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-n729t\"" Nov 03 17:33:41.452810 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452697 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-hndzv\"" Nov 03 17:33:41.452810 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452781 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.453035 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.452896 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Nov 03 17:33:41.453110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453045 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.453110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453045 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-79kzw\"" Nov 03 17:33:41.453110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453060 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Nov 03 17:33:41.453110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453061 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.453110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453097 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-42p6c\"" Nov 03 17:33:41.453291 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453155 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Nov 03 17:33:41.453372 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.453359 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Nov 03 17:33:41.454689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.454670 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.454771 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.454652 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Nov 03 17:33:41.454771 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.454675 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Nov 03 17:33:41.458900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.458880 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-htgnv\"" Nov 03 17:33:41.458986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.458947 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d499d\" (UniqueName: \"kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d\") pod \"migrator-66df5d496b-r2zn7\" (UID: \"cb176382-bf16-4088-9f2e-0ec9932511f5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:41.459056 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.458982 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Nov 03 17:33:41.459056 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.458983 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.459150 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459090 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.459192 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459171 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.459241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459200 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.459241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459221 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-snapshots\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.459312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459236 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.459312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459300 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.459489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459468 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.459603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459499 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sbl5\" (UniqueName: \"kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.459603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459536 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-config\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.459725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459658 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.459809 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459794 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.459841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459823 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.459870 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459849 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.459903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459879 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.459903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459883 4980 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Nov 03 17:33:41.459970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459909 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.459970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459949 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.459970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/048b7350-b701-4e23-856e-67d3413dc370-data-volume\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.460053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.459991 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:41.460053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460025 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.460110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460059 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d69bj\" (UniqueName: \"kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.460110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460093 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.460167 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460117 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8z6h\" (UniqueName: \"kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.460167 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460142 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.460231 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460167 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.460231 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460191 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.460231 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460216 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2hnz\" (UniqueName: \"kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.460356 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460240 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.460356 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460273 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.460447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460433 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zc79m\" (UniqueName: \"kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.460479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460457 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.460479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460474 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.460546 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460489 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2qk9\" (UniqueName: \"kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.460546 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460505 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.460546 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460527 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.460707 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460551 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.460707 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460671 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-config\" (UniqueName: \"kubernetes.io/configmap/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-config\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.460707 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460677 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xddlm\" (UniqueName: \"kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.460838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460708 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.460838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460724 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.460838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460729 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/56cf9464-6022-41a2-b0bc-be87d6b3b90e-serving-cert\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.460838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460739 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.460838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460800 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-trusted-ca\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460852 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460888 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460909 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/645841ea-aa0f-4baa-8923-a460dc301b5a-tmp-dir\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460915 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.460972 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461010 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.461054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461048 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461076 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461069 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461180 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461203 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461222 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8zdt\" (UniqueName: \"kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt\") pod \"kas-connection-checker-685895c7c9-p4zh5\" (UID: \"bea67a96-85b1-41d3-b177-bfec1d287457\") " pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461234 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/3ba272e4-ee5f-4abd-9c93-26f5a2563098-telemetry-config\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461241 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461259 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461274 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461291 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dm74\" (UniqueName: \"kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461305 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.461324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461327 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461384 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461417 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461454 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461490 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461521 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461584 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjp2c\" (UniqueName: \"kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461620 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461675 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q496r\" (UniqueName: \"kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461728 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.461778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461753 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnhsz\" (UniqueName: \"kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.462067 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461923 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.462067 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.461962 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.462067 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462015 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7682\" (UniqueName: \"kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682\") pod \"network-check-source-6f78d59f99-sqcln\" (UID: \"8aa2379e-1eed-4cce-a9df-e96665bf3fd9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:41.462147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462111 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.462147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462134 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwc8l\" (UniqueName: \"kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l\") pod \"kas-connection-checker-685895c7c9-6tgbt\" (UID: \"dfad8e5c-4000-47b2-b39c-ea805fd6e315\") " pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:41.462241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462159 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.462241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462187 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.462322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462244 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.462322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462274 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.462410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462320 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-config\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.462410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462346 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.462410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462375 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.462533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462423 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhgmr\" (UniqueName: \"kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.462533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462447 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.462533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462463 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.462533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462490 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gr8jh\" (UniqueName: \"kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:41.462719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462552 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.462719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462608 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.462719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462637 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:41.462719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462672 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.462719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462695 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462728 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462807 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462822 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462858 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v88mt\" (UniqueName: \"kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt\") pod \"volume-data-source-validator-58c9f5f46d-dfx2h\" (UID: \"5d987bf9-3048-4b49-ba64-010106066297\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462887 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462912 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.462938 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.463023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463007 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463033 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463061 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463085 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463126 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463146 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463162 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463182 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463228 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463238 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cf9464-6022-41a2-b0bc-be87d6b3b90e-tmp\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.463335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463299 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-serving-cert\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.463608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.463363 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-serving-cert\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.494226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.494176 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-vrg7g\"" Nov 03 17:33:41.499459 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.499438 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Nov 03 17:33:41.499557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.499466 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-mnwfr\"" Nov 03 17:33:41.499557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.499475 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.499557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.499475 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Nov 03 17:33:41.500405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500394 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Nov 03 17:33:41.500696 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500681 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.500744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500708 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.500744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500718 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.500744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500714 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.500744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500733 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Nov 03 17:33:41.500890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.500797 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.501154 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.501142 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Nov 03 17:33:41.501592 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.501565 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.502146 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502133 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-stats-auth\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.502647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502596 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Nov 03 17:33:41.502647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502597 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Nov 03 17:33:41.502647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502641 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.502647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502648 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.502834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502674 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.502834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502717 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-lr55j\"" Nov 03 17:33:41.502834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502767 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-jhqc8\"" Nov 03 17:33:41.502977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502949 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Nov 03 17:33:41.503016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.502978 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-konnectivity-ca\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:41.503110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.503093 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-service-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.503556 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.503543 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-default-certificate\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.511112 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.511091 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/56cf9464-6022-41a2-b0bc-be87d6b3b90e-trusted-ca-bundle\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.513355 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.513341 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.520332 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.520311 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d69bj\" (UniqueName: \"kubernetes.io/projected/92e67f15-1027-4a0d-8491-736e2cd4bc04-kube-api-access-d69bj\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.521179 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.521159 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.521743 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.521715 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xddlm\" (UniqueName: \"kubernetes.io/projected/56cf9464-6022-41a2-b0bc-be87d6b3b90e-kube-api-access-xddlm\") pod \"insights-operator-7c4986d449-4x9kc\" (UID: \"56cf9464-6022-41a2-b0bc-be87d6b3b90e\") " pod="openshift-insights/insights-operator-7c4986d449-4x9kc" Nov 03 17:33:41.521895 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.521879 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.521943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.521882 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8zdt\" (UniqueName: \"kubernetes.io/projected/bea67a96-85b1-41d3-b177-bfec1d287457-kube-api-access-c8zdt\") pod \"kas-connection-checker-685895c7c9-p4zh5\" (UID: \"bea67a96-85b1-41d3-b177-bfec1d287457\") " pod="kube-system/kas-connection-checker-685895c7c9-p4zh5" Nov 03 17:33:41.522597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.522559 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjp2c\" (UniqueName: \"kubernetes.io/projected/6a6cf569-649e-420f-8fcf-3c1443782d9e-kube-api-access-zjp2c\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.522637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.522559 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dm74\" (UniqueName: \"kubernetes.io/projected/3ba272e4-ee5f-4abd-9c93-26f5a2563098-kube-api-access-6dm74\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.523112 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.523096 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2qk9\" (UniqueName: \"kubernetes.io/projected/a82bcfea-add1-4518-b0cc-f9b6b9d61f7b-kube-api-access-r2qk9\") pod \"console-operator-7cd56b8dbc-mpbwn\" (UID: \"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b\") " pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:41.524175 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.524159 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zc79m\" (UniqueName: \"kubernetes.io/projected/048b7350-b701-4e23-856e-67d3413dc370-kube-api-access-zc79m\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.525316 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.525300 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v88mt\" (UniqueName: \"kubernetes.io/projected/5d987bf9-3048-4b49-ba64-010106066297-kube-api-access-v88mt\") pod \"volume-data-source-validator-58c9f5f46d-dfx2h\" (UID: \"5d987bf9-3048-4b49-ba64-010106066297\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-58c9f5f46d-dfx2h" Nov 03 17:33:41.525378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.525363 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7682\" (UniqueName: \"kubernetes.io/projected/8aa2379e-1eed-4cce-a9df-e96665bf3fd9-kube-api-access-g7682\") pod \"network-check-source-6f78d59f99-sqcln\" (UID: \"8aa2379e-1eed-4cce-a9df-e96665bf3fd9\") " pod="openshift-network-diagnostics/network-check-source-6f78d59f99-sqcln" Nov 03 17:33:41.525617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.525602 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhgmr\" (UniqueName: \"kubernetes.io/projected/0a59ced0-24a7-45f0-a0c7-de168cbc59a7-kube-api-access-fhgmr\") pod \"service-ca-operator-544d749c8c-jpwjc\" (UID: \"0a59ced0-24a7-45f0-a0c7-de168cbc59a7\") " pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" Nov 03 17:33:41.526355 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.526337 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwc8l\" (UniqueName: \"kubernetes.io/projected/dfad8e5c-4000-47b2-b39c-ea805fd6e315-kube-api-access-hwc8l\") pod \"kas-connection-checker-685895c7c9-6tgbt\" (UID: \"dfad8e5c-4000-47b2-b39c-ea805fd6e315\") " pod="kube-system/kas-connection-checker-685895c7c9-6tgbt" Nov 03 17:33:41.551596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.551561 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.551686 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.551595 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.551686 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.551586 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Nov 03 17:33:41.551686 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.551600 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.551686 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.551667 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552146 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552146 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552166 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552166 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552187 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552196 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552169 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552187 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552198 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552227 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552216 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552196 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552295 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552323 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552332 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552333 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552404 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552458 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552471 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552520 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552557 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552591 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552595 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552700 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552724 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-config\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552749 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552767 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Nov 03 17:33:41.552792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552772 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552805 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552806 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.552937 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553030 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553078 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-gxb7s\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553099 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553119 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553185 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-4tvvh\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553242 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-9ftgc\"" Nov 03 17:33:41.553342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553253 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2pwfx\"" Nov 03 17:33:41.553689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553512 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xxxqk\"" Nov 03 17:33:41.553689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553611 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4xhj6\"" Nov 03 17:33:41.553689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553685 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-8jj5j\"" Nov 03 17:33:41.553796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553700 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Nov 03 17:33:41.553796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553728 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-bstfb\"" Nov 03 17:33:41.553796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553768 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-2vfjt\"" Nov 03 17:33:41.553899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553834 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-2qfzl\"" Nov 03 17:33:41.553899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553855 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553905 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-6jxmj\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553907 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-nx9c7\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553911 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-zfxsl\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553914 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-xhql4\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553929 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-6zl7j\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553964 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Nov 03 17:33:41.553992 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553975 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-glkrd\"" Nov 03 17:33:41.554195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.553999 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-df4kg\"" Nov 03 17:33:41.554195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.554050 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Nov 03 17:33:41.554195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.554104 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-dockercfg-55gxq\"" Nov 03 17:33:41.554287 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.554276 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Nov 03 17:33:41.556636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.556618 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gr8jh\" (UniqueName: \"kubernetes.io/projected/cbea6928-9579-4606-afd8-42fccd905154-kube-api-access-gr8jh\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:41.556916 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.556903 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q496r\" (UniqueName: \"kubernetes.io/projected/92dc67cc-a063-43af-9a22-432e61ee10f4-kube-api-access-q496r\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.557032 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.557019 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnhsz\" (UniqueName: \"kubernetes.io/projected/fe747a0c-5a1d-4db3-876a-f26e1852450e-kube-api-access-rnhsz\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.560153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560135 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/0767163e-8374-4f56-9d2f-e15e26fdcd2e-config-volume\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.560190 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560133 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.560326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560305 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.560435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560416 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d499d\" (UniqueName: \"kubernetes.io/projected/cb176382-bf16-4088-9f2e-0ec9932511f5-kube-api-access-d499d\") pod \"migrator-66df5d496b-r2zn7\" (UID: \"cb176382-bf16-4088-9f2e-0ec9932511f5\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-r2zn7" Nov 03 17:33:41.560481 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560459 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.560522 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560509 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sbl5\" (UniqueName: \"kubernetes.io/projected/d1796bb1-d79a-448a-9152-8570b92e43e6-kube-api-access-7sbl5\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.560599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560561 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-env-overrides\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.560752 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560738 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8z6h\" (UniqueName: \"kubernetes.io/projected/645841ea-aa0f-4baa-8923-a460dc301b5a-kube-api-access-t8z6h\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.560875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.560862 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2hnz\" (UniqueName: \"kubernetes.io/projected/62660df0-cd00-4e15-87ce-7d8177463419-kube-api-access-h2hnz\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.561414 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561397 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cad57e2f-34f9-4ad6-82d7-3ad939d0fef4-agent-certs\") pod \"konnectivity-agent-vwkww\" (UID: \"cad57e2f-34f9-4ad6-82d7-3ad939d0fef4\") " pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:41.561504 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561491 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/62660df0-cd00-4e15-87ce-7d8177463419-serviceca\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.561628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561613 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.561717 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561702 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovn-node-metrics-cert\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.561767 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561722 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.561767 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561755 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.561933 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.561918 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.562562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.562546 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/92dc67cc-a063-43af-9a22-432e61ee10f4-cni-binary-copy\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.562690 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.562673 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fe747a0c-5a1d-4db3-876a-f26e1852450e-ovnkube-script-lib\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.562741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.562687 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.563826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563808 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.563826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563820 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fedfb5ca-94b5-4850-8020-a7279aa49eae-original-pull-secret\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563830 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563847 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563863 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563867 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563878 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563871 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-cnibin\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563911 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563915 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:41.563942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563942 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-systemd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.563956 4980 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.563982 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564023 4980 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564036 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064022033 +0000 UTC m=+1.893105629 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564058 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564073 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064066899 +0000 UTC m=+1.893150492 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564068 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-node-log\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564103 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564135 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564211 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564232 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564248 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpwnq\" (UniqueName: \"kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564263 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564287 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-log-socket\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564311 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8f4f29a5-0dfb-4515-9bd6-276c084934bb-iptables-alerter-script\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564329 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564356 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.564379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564393 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564398 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564435 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564444 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564496 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564506 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-system-cni-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564520 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064508233 +0000 UTC m=+1.893591834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564552 4980 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564551 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564600 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cni-binary-copy\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564614 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064603863 +0000 UTC m=+1.893687443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564641 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564665 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564682 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564682 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-etc-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564695 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564730 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/645841ea-aa0f-4baa-8923-a460dc301b5a-hosts-file\") pod \"node-resolver-gkxfp\" (UID: \"645841ea-aa0f-4baa-8923-a460dc301b5a\") " pod="openshift-dns/node-resolver-gkxfp" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564726 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564752 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564755 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-run-ovn\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564768 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564770 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/048b7350-b701-4e23-856e-67d3413dc370-crio-socket\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564784 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564789 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564801 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564811 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-registration-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564829 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564841 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064829928 +0000 UTC m=+1.893913532 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564867 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564885 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564900 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564916 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564927 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-slash\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564935 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.564966 4980 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564969 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-sys-fs\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564967 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpb8d\" (UniqueName: \"kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564989 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.565004 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.064995116 +0000 UTC m=+1.894078695 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.564965 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-kubelet\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565028 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.565046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565043 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565059 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565131 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565268 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565274 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/2be7ebc7-1f75-4f7b-a18e-7d8103188551-nginx-conf\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565273 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/67333be4-e8fc-47bd-89f3-5873637f8193-cert\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.565267 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.565302 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565302 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565323 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.565333 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.065320582 +0000 UTC m=+1.894404193 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565363 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565366 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-socket-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565383 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565399 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565444 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-etc-selinux\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565448 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565612 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565488 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-var-lib-openvswitch\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565826 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.565885 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txrr7\" (UniqueName: \"kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566135 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566201 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-bin\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566259 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566302 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-695fn\" (UniqueName: \"kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566335 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566367 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.566409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566401 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-999ff\" (UniqueName: \"kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff\") pod \"kas-connection-checker-685895c7c9-w5ndm\" (UID: \"defb7da5-8103-40ef-9607-49c56c281187\") " pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566482 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566515 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566544 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566553 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-config\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566593 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566658 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-systemd-units\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566751 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-tuned\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.566820 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-dbus\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567036 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567094 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567136 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567177 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567188 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-device-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567211 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.567306 4980 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.567355 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.067341215 +0000 UTC m=+1.896424809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567390 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgzgp\" (UniqueName: \"kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567404 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/32a05a82-dc96-44b1-b348-bef87dcc963c-tmp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567426 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567460 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72r2f\" (UniqueName: \"kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567489 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567520 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567553 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.567620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567606 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567636 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567660 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567690 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567721 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567753 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567779 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567814 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567844 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567876 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567906 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567909 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/92dc67cc-a063-43af-9a22-432e61ee10f4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xmdk5\" (UID: \"92dc67cc-a063-43af-9a22-432e61ee10f4\") " pod="openshift-multus/multus-additional-cni-plugins-xmdk5" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568006 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d1796bb1-d79a-448a-9152-8570b92e43e6-kubelet-dir\") pod \"aws-ebs-csi-driver-node-z86jm\" (UID: \"d1796bb1-d79a-448a-9152-8570b92e43e6\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-z86jm" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568020 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-cni-netd\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568090 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0767163e-8374-4f56-9d2f-e15e26fdcd2e-tmp-dir\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568100 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/62660df0-cd00-4e15-87ce-7d8177463419-host\") pod \"node-ca-gv7lt\" (UID: \"62660df0-cd00-4e15-87ce-7d8177463419\") " pod="openshift-image-registry/node-ca-gv7lt" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568158 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fedfb5ca-94b5-4850-8020-a7279aa49eae-kubelet-config\") pod \"global-pull-secret-syncer-fnq58\" (UID: \"fedfb5ca-94b5-4850-8020-a7279aa49eae\") " pod="kube-system/global-pull-secret-syncer-fnq58" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568209 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe747a0c-5a1d-4db3-876a-f26e1852450e-host-run-netns\") pod \"ovnkube-node-d8hsc\" (UID: \"fe747a0c-5a1d-4db3-876a-f26e1852450e\") " pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.568300 4980 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.567842 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-serving-cert\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.568317 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-daemon-config\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.568381 4980 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.568438 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.068422817 +0000 UTC m=+1.897506408 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : secret "metrics-daemon-secret" not found Nov 03 17:33:41.568722 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.568484 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.068473616 +0000 UTC m=+1.897557215 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:41.624969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.624940 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpwnq\" (UniqueName: \"kubernetes.io/projected/0767163e-8374-4f56-9d2f-e15e26fdcd2e-kube-api-access-xpwnq\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:41.625142 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.625123 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b22p\" (UniqueName: \"kubernetes.io/projected/333b0df1-e4ba-4064-979c-655f7e52b9f0-kube-api-access-9b22p\") pod \"network-check-target-xp42v\" (UID: \"333b0df1-e4ba-4064-979c-655f7e52b9f0\") " pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:41.625199 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.625126 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpb8d\" (UniqueName: \"kubernetes.io/projected/67333be4-e8fc-47bd-89f3-5873637f8193-kube-api-access-jpb8d\") pod \"ingress-canary-rtv8j\" (UID: \"67333be4-e8fc-47bd-89f3-5873637f8193\") " pod="openshift-ingress-canary/ingress-canary-rtv8j" Nov 03 17:33:41.626002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.625986 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txrr7\" (UniqueName: \"kubernetes.io/projected/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-kube-api-access-txrr7\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.627737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.627720 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-695fn\" (UniqueName: \"kubernetes.io/projected/8f4f29a5-0dfb-4515-9bd6-276c084934bb-kube-api-access-695fn\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.627782 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.627734 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-999ff\" (UniqueName: \"kubernetes.io/projected/defb7da5-8103-40ef-9607-49c56c281187-kube-api-access-999ff\") pod \"kas-connection-checker-685895c7c9-w5ndm\" (UID: \"defb7da5-8103-40ef-9607-49c56c281187\") " pod="kube-system/kas-connection-checker-685895c7c9-w5ndm" Nov 03 17:33:41.627827 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.627813 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgzgp\" (UniqueName: \"kubernetes.io/projected/32a05a82-dc96-44b1-b348-bef87dcc963c-kube-api-access-kgzgp\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.651237 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.651213 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72r2f\" (UniqueName: \"kubernetes.io/projected/9584d3a2-671c-48e3-89f9-dfd8e38db4ab-kube-api-access-72r2f\") pod \"kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn\" (UID: \"9584d3a2-671c-48e3-89f9-dfd8e38db4ab\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" Nov 03 17:33:41.668359 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668337 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668434 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668361 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668434 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668388 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.668434 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668403 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668434 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668420 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668436 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668445 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-k8s-cni-cncf-io\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668456 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668471 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8f4f29a5-0dfb-4515-9bd6-276c084934bb-host-slash\") pod \"iptables-alerter-hp94c\" (UID: \"8f4f29a5-0dfb-4515-9bd6-276c084934bb\") " pod="openshift-network-operator/iptables-alerter-hp94c" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668485 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668498 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-sys\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668508 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-netns\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668448 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-kubelet\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668515 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668537 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-socket-dir-parent\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668550 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668553 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-bin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668592 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668604 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-conf\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668627 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysconfig\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668631 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668660 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668684 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.668709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668717 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-lib-modules\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668732 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668770 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668793 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-modprobe-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668801 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-sysctl-d\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668822 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668830 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-var-lib-cni-multus\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668840 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-etc-kubernetes\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668846 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668863 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-cnibin\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668881 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-system-cni-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668910 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668947 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668964 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668985 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-var-lib-kubelet\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668983 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-run\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669005 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.668985 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-multus-conf-dir\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669015 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-systemd\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669066 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669110 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669134 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669155 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-hostroot\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669162 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669180 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-host\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669157 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/32a05a82-dc96-44b1-b348-bef87dcc963c-etc-kubernetes\") pod \"tuned-8jlfb\" (UID: \"32a05a82-dc96-44b1-b348-bef87dcc963c\") " pod="openshift-cluster-node-tuning-operator/tuned-8jlfb" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669189 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.669210 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a7f26304-7fb0-49b7-8f59-d8dc02c88d09-host-run-multus-certs\") pod \"multus-knk2x\" (UID: \"a7f26304-7fb0-49b7-8f59-d8dc02c88d09\") " pod="openshift-multus/multus-knk2x" Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.669229 4980 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:41.669458 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:41.669311 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:42.1692942 +0000 UTC m=+1.998377794 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:41.699062 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.699046 4980 scope.go:122] "RemoveContainer" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" Nov 03 17:33:41.955362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:41.955336 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:42.072747 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072675 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:42.072747 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072712 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072747 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072776 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072806 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072830 4980 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072860 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072879 4980 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072830 4980 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072899 4980 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:42.072925 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072903 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.072886442 +0000 UTC m=+2.901970033 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : secret "metrics-daemon-secret" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.072952 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072969 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.072959125 +0000 UTC m=+2.902042722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072971 4980 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072996 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.072987716 +0000 UTC m=+2.902071309 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.072999 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073008 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073013 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073005406 +0000 UTC m=+2.902088998 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073045 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073036833 +0000 UTC m=+2.902120416 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073062 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073056144 +0000 UTC m=+2.902139727 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073077 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073072242 +0000 UTC m=+2.902155826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.073115 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.073138 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.073182 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073230 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073237 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073240 4980 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Nov 03 17:33:42.073245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073252 4980 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:42.073761 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073269 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073258377 +0000 UTC m=+2.902341977 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:42.073761 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073286 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.07327605 +0000 UTC m=+2.902359647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:42.073761 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.073301 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.073294801 +0000 UTC m=+2.902378384 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : configmap "kube-rbac-proxy" not found Nov 03 17:33:42.174380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.174351 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:42.174519 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.174480 4980 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:42.174557 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.174534 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:43.174520559 +0000 UTC m=+3.003604153 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:42.350400 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.350329 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:33:42.350812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.350745 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/1.log" Nov 03 17:33:42.350812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.350774 4980 generic.go:356] "Generic (PLEG): container finished" podID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerID="4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23" exitCode=255 Nov 03 17:33:42.350894 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.350868 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerDied","Data":"4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23"} Nov 03 17:33:42.350952 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.350936 4980 scope.go:122] "RemoveContainer" containerID="16966937d4394cc389dfe9cf866e095881e8a6464445782a856c428f30f1aca0" Nov 03 17:33:42.351734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.351715 4980 scope.go:122] "RemoveContainer" containerID="4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23" Nov 03 17:33:42.351943 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:42.351885 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:42.490510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:42.490483 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:33:43.082749 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.082706 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:43.082926 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.082903 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:43.082980 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.082945 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:43.083051 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083028 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:43.083103 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083066 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:43.083152 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083104 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:43.083217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083200 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:43.083267 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083243 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:43.083325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083278 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:43.083367 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.083338 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:43.083511 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.083489 4980 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:43.083595 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.083562 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.083542869 +0000 UTC m=+4.912626475 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:43.083678 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.083665 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:43.083748 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.083681 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:43.083794 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.083762 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.083750781 +0000 UTC m=+4.912834387 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:43.084318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.084295 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/048b7350-b701-4e23-856e-67d3413dc370-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:43.084444 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084429 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:43.084512 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084446 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:43.084512 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084493 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.084479784 +0000 UTC m=+4.913563384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:43.084627 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084587 4980 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:43.084672 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084629 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.084618662 +0000 UTC m=+4.913702257 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:43.084719 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084698 4980 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 17:33:43.084800 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084774 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.084762241 +0000 UTC m=+4.913845835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : secret "metrics-daemon-secret" not found Nov 03 17:33:43.084938 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084925 4980 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:43.085028 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.084970 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.084959161 +0000 UTC m=+4.914042744 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:43.085093 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.085076 4980 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:43.085141 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.085125 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.085113706 +0000 UTC m=+4.914197303 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:43.085210 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.085196 4980 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:43.085256 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.085243 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.085226324 +0000 UTC m=+4.914309920 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:43.085368 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.085357 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.085347075 +0000 UTC m=+4.914430669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:43.184707 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.184681 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:43.184841 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.184795 4980 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:43.184841 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:43.184836 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:45.184825936 +0000 UTC m=+5.013909529 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:43.355792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:43.355725 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:33:45.101637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101599 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:45.101637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101644 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101663 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101744 4980 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101745 4980 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101769 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101745 4980 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101801 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.101788421 +0000 UTC m=+8.930872001 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101846 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.101829298 +0000 UTC m=+8.930912890 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : secret "router-metrics-certs-default" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101863 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.101855904 +0000 UTC m=+8.930939495 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.101877 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.101871171 +0000 UTC m=+8.930954755 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101901 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101930 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.101963 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.102027 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102046 4980 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.102059 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102067 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102086 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102098 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.102086249 +0000 UTC m=+8.931169846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102118 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102119 4980 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102129 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102138 4980 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102127 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.102112795 +0000 UTC m=+8.931196393 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:45.102157 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102175 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.102165218 +0000 UTC m=+8.931248799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:45.102911 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102185 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.102180044 +0000 UTC m=+8.931263624 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:45.102911 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.102196 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs podName:cbea6928-9579-4606-afd8-42fccd905154 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.102191079 +0000 UTC m=+8.931274659 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs") pod "network-metrics-daemon-tlq8b" (UID: "cbea6928-9579-4606-afd8-42fccd905154") : secret "metrics-daemon-secret" not found Nov 03 17:33:45.203025 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:45.202997 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:45.203177 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.203116 4980 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Nov 03 17:33:45.203177 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:45.203160 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert podName:2be7ebc7-1f75-4f7b-a18e-7d8103188551 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:49.203149609 +0000 UTC m=+9.032233189 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert") pod "networking-console-plugin-86c478bb88-gcgd8" (UID: "2be7ebc7-1f75-4f7b-a18e-7d8103188551") : secret "networking-console-plugin-cert" not found Nov 03 17:33:46.167352 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:46.167322 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:46.167795 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:46.167522 4980 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:33:48.222917 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:48.222884 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-d8hsc" Nov 03 17:33:48.556527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:48.556451 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gkxfp_645841ea-aa0f-4baa-8923-a460dc301b5a/dns-node-resolver/0.log" Nov 03 17:33:49.139119 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139080 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139142 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139166 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139187 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139209 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139234 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139241 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139263 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-59985d89dd-5th7m: secret "image-registry-tls" not found Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139305 4980 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Nov 03 17:33:49.139340 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139325 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls podName:9d87a0d2-2d79-4cff-8a36-c8f3f6580b94 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139307353 +0000 UTC m=+16.968390949 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls") pod "image-registry-59985d89dd-5th7m" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94") : secret "image-registry-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139341 4980 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139353 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle podName:6a6cf569-649e-420f-8fcf-3c1443782d9e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139336833 +0000 UTC m=+16.968420428 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle") pod "router-default-5b97b666bc-r5q4p" (UID: "6a6cf569-649e-420f-8fcf-3c1443782d9e") : configmap references non-existent config key: service-ca.crt Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139261 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139367 4980 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139371 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls podName:0767163e-8374-4f56-9d2f-e15e26fdcd2e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139363401 +0000 UTC m=+16.968446990 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls") pod "dns-default-chggk" (UID: "0767163e-8374-4f56-9d2f-e15e26fdcd2e") : secret "dns-default-metrics-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139464 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls podName:048b7350-b701-4e23-856e-67d3413dc370 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139423529 +0000 UTC m=+16.968507113 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls") pod "insights-runtime-extractor-42zww" (UID: "048b7350-b701-4e23-856e-67d3413dc370") : secret "insights-runtime-extractor-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139494 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls podName:3ba272e4-ee5f-4abd-9c93-26f5a2563098 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139482378 +0000 UTC m=+16.968565971 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-884576c57-zzdc8" (UID: "3ba272e4-ee5f-4abd-9c93-26f5a2563098") : secret "cluster-monitoring-operator-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139549 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139598 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.139600 4980 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139663 4980 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139676 4980 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139688 4980 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6b4c9976fc-ckgdt: secret "image-registry-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139703 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls podName:92e67f15-1027-4a0d-8491-736e2cd4bc04 nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139690851 +0000 UTC m=+16.968774453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls") pod "cluster-samples-operator-6ddb6c8b94-l5lgj" (UID: "92e67f15-1027-4a0d-8491-736e2cd4bc04") : secret "samples-operator-tls" not found Nov 03 17:33:49.139796 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.139748 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls podName:5f42720f-0f0f-4b89-a6e3-6c4b0e42260e nodeName:}" failed. No retries permitted until 2026-11-03 17:33:57.139733486 +0000 UTC m=+16.968817094 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls") pod "image-registry-6b4c9976fc-ckgdt" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e") : secret "image-registry-tls" not found Nov 03 17:33:49.141945 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.141926 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cbea6928-9579-4606-afd8-42fccd905154-metrics-certs\") pod \"network-metrics-daemon-tlq8b\" (UID: \"cbea6928-9579-4606-afd8-42fccd905154\") " pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:49.142040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.141980 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a6cf569-649e-420f-8fcf-3c1443782d9e-metrics-certs\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:49.205256 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.205230 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tlq8b" Nov 03 17:33:49.240682 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.240637 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:49.243835 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.243813 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/2be7ebc7-1f75-4f7b-a18e-7d8103188551-networking-console-plugin-cert\") pod \"networking-console-plugin-86c478bb88-gcgd8\" (UID: \"2be7ebc7-1f75-4f7b-a18e-7d8103188551\") " pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:49.322664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.322631 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:49.323016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.323003 4980 scope.go:122] "RemoveContainer" containerID="4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23" Nov 03 17:33:49.323205 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:49.323189 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:49.504840 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.504808 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" Nov 03 17:33:49.556970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.556939 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-gv7lt_62660df0-cd00-4e15-87ce-7d8177463419/node-ca/0.log" Nov 03 17:33:49.577747 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.577717 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tlq8b"] Nov 03 17:33:49.581500 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:49.581472 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbea6928_9579_4606_afd8_42fccd905154.slice/crio-45ba288fe019169517a24a02c2bebedffcca5e81abf22b1c1e812d7529520d82 WatchSource:0}: Error finding container 45ba288fe019169517a24a02c2bebedffcca5e81abf22b1c1e812d7529520d82: Status 404 returned error can't find the container with id 45ba288fe019169517a24a02c2bebedffcca5e81abf22b1c1e812d7529520d82 Nov 03 17:33:49.583720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.583696 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:33:49.876148 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:49.876070 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-86c478bb88-gcgd8"] Nov 03 17:33:49.879084 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:49.879055 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2be7ebc7_1f75_4f7b_a18e_7d8103188551.slice/crio-b5eac5743dea857943bfee873cb5baa58154362f8b62108c9c5e11053862dfbd WatchSource:0}: Error finding container b5eac5743dea857943bfee873cb5baa58154362f8b62108c9c5e11053862dfbd: Status 404 returned error can't find the container with id b5eac5743dea857943bfee873cb5baa58154362f8b62108c9c5e11053862dfbd Nov 03 17:33:50.326145 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.326117 4980 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:50.326533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.326470 4980 scope.go:122] "RemoveContainer" containerID="4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23" Nov 03 17:33:50.326733 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:33:50.326710 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7cd56b8dbc-mpbwn_openshift-console-operator(a82bcfea-add1-4518-b0cc-f9b6b9d61f7b)\"" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" Nov 03 17:33:50.355489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.355465 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rtv8j_67333be4-e8fc-47bd-89f3-5873637f8193/serve-healthcheck-canary/0.log" Nov 03 17:33:50.380004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.379977 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" event={"ID":"2be7ebc7-1f75-4f7b-a18e-7d8103188551","Type":"ContainerStarted","Data":"b5eac5743dea857943bfee873cb5baa58154362f8b62108c9c5e11053862dfbd"} Nov 03 17:33:50.381129 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.381110 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlq8b" event={"ID":"cbea6928-9579-4606-afd8-42fccd905154","Type":"ContainerStarted","Data":"45ba288fe019169517a24a02c2bebedffcca5e81abf22b1c1e812d7529520d82"} Nov 03 17:33:50.556645 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.556619 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-r2zn7_cb176382-bf16-4088-9f2e-0ec9932511f5/migrator/0.log" Nov 03 17:33:50.957421 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:50.957391 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn_9584d3a2-671c-48e3-89f9-dfd8e38db4ab/kube-storage-version-migrator-operator/0.log" Nov 03 17:33:51.049877 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:51.049847 4980 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-139-234.us-west-1.compute.internal" event="NodeReady" Nov 03 17:33:51.386351 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:51.386325 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlq8b" event={"ID":"cbea6928-9579-4606-afd8-42fccd905154","Type":"ContainerStarted","Data":"8ac3e317d39763c0453b63888c2a300ba57f325e0b4dcdbc4caf6f698f940a4a"} Nov 03 17:33:51.386711 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:51.386372 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tlq8b" event={"ID":"cbea6928-9579-4606-afd8-42fccd905154","Type":"ContainerStarted","Data":"0abdb9fe96edc3bb8f7be67c1d172d486330bf8f385d97eabbdcd79c7bd1ac95"} Nov 03 17:33:52.291243 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.291219 4980 scope.go:122] "RemoveContainer" containerID="4a8c4e359cdcfcd78040d6a9c6b811c9e645bec2a11296a34d09321a112d3a23" Nov 03 17:33:52.391926 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.391893 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:33:52.392288 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.392012 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" event={"ID":"a82bcfea-add1-4518-b0cc-f9b6b9d61f7b","Type":"ContainerStarted","Data":"5f8f2c013d918a7b6001bab1e300b0695c8d44f0af8d96512dd899b407068344"} Nov 03 17:33:52.392608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.392542 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:52.393089 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.393058 4980 patch_prober.go:29] interesting pod/console-operator-7cd56b8dbc-mpbwn container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" start-of-body= Nov 03 17:33:52.393301 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.393102 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" podUID="a82bcfea-add1-4518-b0cc-f9b6b9d61f7b" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.14:8443/readyz\": dial tcp 10.132.0.14:8443: connect: connection refused" Nov 03 17:33:52.394007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:52.393985 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-86c478bb88-gcgd8" event={"ID":"2be7ebc7-1f75-4f7b-a18e-7d8103188551","Type":"ContainerStarted","Data":"5fdab9bceec6817ddb9ca9a96978ce74f86e7a41b5e7899c25d47ec6a5ba4dcb"} Nov 03 17:33:53.468985 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:53.468948 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7cd56b8dbc-mpbwn" Nov 03 17:33:57.208665 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.208623 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:57.209060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.208676 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:57.209060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.208797 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:57.209060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.208837 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:57.209060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.208981 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:57.209060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.209029 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:57.209325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.209069 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:57.209774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.209749 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6a6cf569-649e-420f-8fcf-3c1443782d9e-service-ca-bundle\") pod \"router-default-5b97b666bc-r5q4p\" (UID: \"6a6cf569-649e-420f-8fcf-3c1443782d9e\") " pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:57.211628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.211596 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/0767163e-8374-4f56-9d2f-e15e26fdcd2e-metrics-tls\") pod \"dns-default-chggk\" (UID: \"0767163e-8374-4f56-9d2f-e15e26fdcd2e\") " pod="openshift-dns/dns-default-chggk" Nov 03 17:33:57.211880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.211852 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/048b7350-b701-4e23-856e-67d3413dc370-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-42zww\" (UID: \"048b7350-b701-4e23-856e-67d3413dc370\") " pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:57.212136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.212106 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92e67f15-1027-4a0d-8491-736e2cd4bc04-samples-operator-tls\") pod \"cluster-samples-operator-6ddb6c8b94-l5lgj\" (UID: \"92e67f15-1027-4a0d-8491-736e2cd4bc04\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:57.212236 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.212117 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"image-registry-59985d89dd-5th7m\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:57.212236 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.212173 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"image-registry-6b4c9976fc-ckgdt\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:57.212348 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.212277 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/3ba272e4-ee5f-4abd-9c93-26f5a2563098-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-884576c57-zzdc8\" (UID: \"3ba272e4-ee5f-4abd-9c93-26f5a2563098\") " pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:57.297815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.297793 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" Nov 03 17:33:57.297919 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.297892 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" Nov 03 17:33:57.302683 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.302666 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:57.303748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.303683 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:57.311561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.311250 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-chggk" Nov 03 17:33:57.311561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.311475 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:57.312304 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.312285 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-42zww" Nov 03 17:33:57.703369 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.703341 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8"] Nov 03 17:33:57.742855 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.742829 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ba272e4_ee5f_4abd_9c93_26f5a2563098.slice/crio-ec4ceb5c7dc321e09be3961b1573282afbd43f43ea43697dcc55053c443e2dfc WatchSource:0}: Error finding container ec4ceb5c7dc321e09be3961b1573282afbd43f43ea43697dcc55053c443e2dfc: Status 404 returned error can't find the container with id ec4ceb5c7dc321e09be3961b1573282afbd43f43ea43697dcc55053c443e2dfc Nov 03 17:33:57.753126 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.753099 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj"] Nov 03 17:33:57.753792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.753770 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:33:57.779557 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.779486 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f42720f_0f0f_4b89_a6e3_6c4b0e42260e.slice/crio-4635934e4ed7b961a709b5a31ef700ee03dc06680eb64c5989547434610e7047 WatchSource:0}: Error finding container 4635934e4ed7b961a709b5a31ef700ee03dc06680eb64c5989547434610e7047: Status 404 returned error can't find the container with id 4635934e4ed7b961a709b5a31ef700ee03dc06680eb64c5989547434610e7047 Nov 03 17:33:57.790849 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.790826 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-chggk"] Nov 03 17:33:57.794613 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.794522 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0767163e_8374_4f56_9d2f_e15e26fdcd2e.slice/crio-92bac81dc47f6ec3daec6ab8f087d164e466f6258be12e9acdffbccede567263 WatchSource:0}: Error finding container 92bac81dc47f6ec3daec6ab8f087d164e466f6258be12e9acdffbccede567263: Status 404 returned error can't find the container with id 92bac81dc47f6ec3daec6ab8f087d164e466f6258be12e9acdffbccede567263 Nov 03 17:33:57.797796 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.797731 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a6cf569_649e_420f_8fcf_3c1443782d9e.slice/crio-8381f226b35307c5899eb732f3c580fa628907122c37047de47daa2d4ef6ccb6 WatchSource:0}: Error finding container 8381f226b35307c5899eb732f3c580fa628907122c37047de47daa2d4ef6ccb6: Status 404 returned error can't find the container with id 8381f226b35307c5899eb732f3c580fa628907122c37047de47daa2d4ef6ccb6 Nov 03 17:33:57.800062 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.800037 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048b7350_b701_4e23_856e_67d3413dc370.slice/crio-f87ed500361df80ff78f7a930df10eb8cca825533ad58cf3490fb3ade896bd67 WatchSource:0}: Error finding container f87ed500361df80ff78f7a930df10eb8cca825533ad58cf3490fb3ade896bd67: Status 404 returned error can't find the container with id f87ed500361df80ff78f7a930df10eb8cca825533ad58cf3490fb3ade896bd67 Nov 03 17:33:57.803345 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.803325 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-42zww"] Nov 03 17:33:57.804731 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.804704 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-5b97b666bc-r5q4p"] Nov 03 17:33:57.805886 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:57.805869 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:33:57.831615 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:33:57.831558 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d87a0d2_2d79_4cff_8a36_c8f3f6580b94.slice/crio-e47648405286276dafebf4f2889d57d2ccdbdaa76029a9c409ff89f59fb9a5cb WatchSource:0}: Error finding container e47648405286276dafebf4f2889d57d2ccdbdaa76029a9c409ff89f59fb9a5cb: Status 404 returned error can't find the container with id e47648405286276dafebf4f2889d57d2ccdbdaa76029a9c409ff89f59fb9a5cb Nov 03 17:33:58.082514 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.082437 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-vwkww" Nov 03 17:33:58.417600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.417504 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" event={"ID":"3ba272e4-ee5f-4abd-9c93-26f5a2563098","Type":"ContainerStarted","Data":"ec4ceb5c7dc321e09be3961b1573282afbd43f43ea43697dcc55053c443e2dfc"} Nov 03 17:33:58.419160 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.419132 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" event={"ID":"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e","Type":"ContainerStarted","Data":"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054"} Nov 03 17:33:58.419291 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.419177 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" event={"ID":"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e","Type":"ContainerStarted","Data":"4635934e4ed7b961a709b5a31ef700ee03dc06680eb64c5989547434610e7047"} Nov 03 17:33:58.419291 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.419242 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:33:58.420429 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.420406 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" event={"ID":"92e67f15-1027-4a0d-8491-736e2cd4bc04","Type":"ContainerStarted","Data":"e16c3217edd876ab80c182f5cdeb9e8f7f5411a6f64b3bea42df1caed40d378d"} Nov 03 17:33:58.421871 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.421845 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" event={"ID":"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94","Type":"ContainerStarted","Data":"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca"} Nov 03 17:33:58.421962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.421876 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" event={"ID":"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94","Type":"ContainerStarted","Data":"e47648405286276dafebf4f2889d57d2ccdbdaa76029a9c409ff89f59fb9a5cb"} Nov 03 17:33:58.421962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.421893 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:33:58.423318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.423299 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42zww" event={"ID":"048b7350-b701-4e23-856e-67d3413dc370","Type":"ContainerStarted","Data":"abec0ef1134852951e0950582a56b72e41392ddaf32d7f6b13c6869efd0de69b"} Nov 03 17:33:58.423411 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.423340 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42zww" event={"ID":"048b7350-b701-4e23-856e-67d3413dc370","Type":"ContainerStarted","Data":"f87ed500361df80ff78f7a930df10eb8cca825533ad58cf3490fb3ade896bd67"} Nov 03 17:33:58.424371 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.424353 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chggk" event={"ID":"0767163e-8374-4f56-9d2f-e15e26fdcd2e","Type":"ContainerStarted","Data":"92bac81dc47f6ec3daec6ab8f087d164e466f6258be12e9acdffbccede567263"} Nov 03 17:33:58.425706 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.425687 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" event={"ID":"6a6cf569-649e-420f-8fcf-3c1443782d9e","Type":"ContainerStarted","Data":"64740e1b2b0363dc17d5cd1195a1b5ac70e08b5db6694d9b9d7cb1cb35347752"} Nov 03 17:33:58.425793 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:58.425713 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" event={"ID":"6a6cf569-649e-420f-8fcf-3c1443782d9e","Type":"ContainerStarted","Data":"8381f226b35307c5899eb732f3c580fa628907122c37047de47daa2d4ef6ccb6"} Nov 03 17:33:59.303926 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:59.303839 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:59.307011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:59.306984 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:59.436585 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:59.436518 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42zww" event={"ID":"048b7350-b701-4e23-856e-67d3413dc370","Type":"ContainerStarted","Data":"aa853bc6f9a63a4798b36a73ec5d761eda6d60b40aa7af7604d42f22216ccb29"} Nov 03 17:33:59.437566 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:59.437544 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:33:59.439165 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:33:59.439144 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-5b97b666bc-r5q4p" Nov 03 17:34:00.450784 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:00.450749 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chggk" event={"ID":"0767163e-8374-4f56-9d2f-e15e26fdcd2e","Type":"ContainerStarted","Data":"180ccc8ca45b179cc4283e391e5e474538dd6a4e347d40d96cd327b3c2699e6b"} Nov 03 17:34:00.454876 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:00.454851 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-884576c57-zzdc8" event={"ID":"3ba272e4-ee5f-4abd-9c93-26f5a2563098","Type":"ContainerStarted","Data":"12c3de703315c76f34298d5d6be0c0556f15ef7e270f897a600eab8b48f5367f"} Nov 03 17:34:00.463414 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:00.463391 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" event={"ID":"92e67f15-1027-4a0d-8491-736e2cd4bc04","Type":"ContainerStarted","Data":"c7a70fb998d08c6aa98c844267ba615bb56db6c948a5617cc2d6e088771638fe"} Nov 03 17:34:01.194818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:01.194788 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:34:01.469600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:01.469543 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-42zww" event={"ID":"048b7350-b701-4e23-856e-67d3413dc370","Type":"ContainerStarted","Data":"d3777ebee2c3bfc302b1815a189053f34b7ce7ed2c9217909aaed872ae7d30e2"} Nov 03 17:34:01.471561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:01.471534 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-chggk" event={"ID":"0767163e-8374-4f56-9d2f-e15e26fdcd2e","Type":"ContainerStarted","Data":"510e1991aacb3e3e7a6a392669cdc5ef9c722f84e027636d0bb48a893b3c9d52"} Nov 03 17:34:01.471697 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:01.471605 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-chggk" Nov 03 17:34:01.473411 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:01.473384 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6ddb6c8b94-l5lgj" event={"ID":"92e67f15-1027-4a0d-8491-736e2cd4bc04","Type":"ContainerStarted","Data":"3a66ae8180a5aeca1d05aac2d93ec360a828104d87debd6678aba6de585198c9"} Nov 03 17:34:11.480041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:11.480004 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-chggk" Nov 03 17:34:12.493175 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:12.493143 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-xp42v" Nov 03 17:34:14.126181 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.126146 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-8vcdj"] Nov 03 17:34:14.150565 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.150533 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246391 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246358 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246396 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-tls\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246415 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-wtmp\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246485 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-root\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246507 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-sys\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246530 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkj46\" (UniqueName: \"kubernetes.io/projected/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-kube-api-access-qkj46\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246611 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-metrics-client-ca\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246636 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-textfile\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.246756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.246738 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.262610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.262557 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Nov 03 17:34:14.262736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.262563 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n8sjq\"" Nov 03 17:34:14.262736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.262592 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Nov 03 17:34:14.262736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.262644 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Nov 03 17:34:14.312339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.312313 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Nov 03 17:34:14.348119 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348085 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348130 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-tls\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348245 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-wtmp\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348293 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-root\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348316 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-sys\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348347 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkj46\" (UniqueName: \"kubernetes.io/projected/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-kube-api-access-qkj46\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348386 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-wtmp\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348403 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-root\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348414 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-sys\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348447 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-metrics-client-ca\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348476 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-textfile\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348529 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348989 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.348855 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-textfile\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.348989 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.349019 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-metrics-client-ca\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.349154 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.349080 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-accelerators-collector-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.350988 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.350920 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-tls\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.351401 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.351376 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.412730 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.412676 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkj46\" (UniqueName: \"kubernetes.io/projected/187d0aaf-1c85-42e0-be08-6bd41ffa9bab-kube-api-access-qkj46\") pod \"node-exporter-8vcdj\" (UID: \"187d0aaf-1c85-42e0-be08-6bd41ffa9bab\") " pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.504044 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:14.504014 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-8vcdj" Nov 03 17:34:14.519321 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:34:14.519289 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:34:14.535976 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:34:14.535951 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod187d0aaf_1c85_42e0_be08_6bd41ffa9bab.slice/crio-a468f30ca4b782cfef3828553602b83e6b89c47bdb56d4ff55bc947b7557d114 WatchSource:0}: Error finding container a468f30ca4b782cfef3828553602b83e6b89c47bdb56d4ff55bc947b7557d114: Status 404 returned error can't find the container with id a468f30ca4b782cfef3828553602b83e6b89c47bdb56d4ff55bc947b7557d114 Nov 03 17:34:15.527471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:15.527427 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vcdj" event={"ID":"187d0aaf-1c85-42e0-be08-6bd41ffa9bab","Type":"ContainerStarted","Data":"a468f30ca4b782cfef3828553602b83e6b89c47bdb56d4ff55bc947b7557d114"} Nov 03 17:34:16.533810 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:16.533773 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vcdj" event={"ID":"187d0aaf-1c85-42e0-be08-6bd41ffa9bab","Type":"ContainerStarted","Data":"19f45b182e0702991cbdeab0b060e155b23d382a7964ac804a5c8ad4f9f0f6ff"} Nov 03 17:34:17.315836 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:17.315802 4980 patch_prober.go:29] interesting pod/image-registry-59985d89dd-5th7m container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Nov 03 17:34:17.316095 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:17.315855 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:34:17.539290 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:17.539256 4980 generic.go:356] "Generic (PLEG): container finished" podID="187d0aaf-1c85-42e0-be08-6bd41ffa9bab" containerID="19f45b182e0702991cbdeab0b060e155b23d382a7964ac804a5c8ad4f9f0f6ff" exitCode=0 Nov 03 17:34:17.539688 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:17.539332 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vcdj" event={"ID":"187d0aaf-1c85-42e0-be08-6bd41ffa9bab","Type":"ContainerDied","Data":"19f45b182e0702991cbdeab0b060e155b23d382a7964ac804a5c8ad4f9f0f6ff"} Nov 03 17:34:18.545087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:18.545050 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vcdj" event={"ID":"187d0aaf-1c85-42e0-be08-6bd41ffa9bab","Type":"ContainerStarted","Data":"08285a89d1f76cf11debdff2d6d89467a211771a312e0f91afe6b051966dec11"} Nov 03 17:34:18.545483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:18.545115 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-8vcdj" event={"ID":"187d0aaf-1c85-42e0-be08-6bd41ffa9bab","Type":"ContainerStarted","Data":"586104bca471aa4c891db20898fb93b2d592bba5d0629f155891babdc0e60800"} Nov 03 17:34:18.709864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:18.709805 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-8vcdj" podStartSLOduration=2.835274736 podStartE2EDuration="4.70979078s" totalImagesPullingTime="1.79235238s" totalInitContainerRuntime="82.163664ms" isStatefulPod=true podCreationTimestamp="2026-11-03 17:34:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:34:18.709194766 +0000 UTC m=+38.538278369" watchObservedRunningTime="2026-11-03 17:34:18.70979078 +0000 UTC m=+38.538874383" Nov 03 17:34:19.440816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:19.440790 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:34:21.200917 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:21.200840 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:34:24.071563 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:24.071528 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:34:26.215605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.215528 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" podUID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" containerName="registry" containerID="cri-o://993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054" gracePeriod=30 Nov 03 17:34:26.446143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.446121 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:34:26.548400 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548338 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548400 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548383 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548408 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548428 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548465 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548494 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548549 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548597 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") pod \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\" (UID: \"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e\") " Nov 03 17:34:26.548948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.548928 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:26.549151 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.549125 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:26.551214 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.551181 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:26.551325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.551307 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:26.551417 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.551385 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:26.551477 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.551418 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:26.551760 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.551743 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "kube-api-access-ntgqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:26.555838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.555815 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted" pod "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" (UID: "5f42720f-0f0f-4b89-a6e3-6c4b0e42260e"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:26.579839 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.579811 4980 generic.go:356] "Generic (PLEG): container finished" podID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" containerID="993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054" exitCode=0 Nov 03 17:34:26.579946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.579859 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" Nov 03 17:34:26.579946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.579858 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" event={"ID":"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e","Type":"ContainerDied","Data":"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054"} Nov 03 17:34:26.580023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.579967 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6b4c9976fc-ckgdt" event={"ID":"5f42720f-0f0f-4b89-a6e3-6c4b0e42260e","Type":"ContainerDied","Data":"4635934e4ed7b961a709b5a31ef700ee03dc06680eb64c5989547434610e7047"} Nov 03 17:34:26.580023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.579992 4980 scope.go:122] "RemoveContainer" containerID="993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054" Nov 03 17:34:26.587975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.587957 4980 scope.go:122] "RemoveContainer" containerID="993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054" Nov 03 17:34:26.588245 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:34:26.588228 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054\": container with ID starting with 993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054 not found: ID does not exist" containerID="993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054" Nov 03 17:34:26.588281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.588254 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054"} err="failed to get container status \"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054\": rpc error: code = NotFound desc = could not find container \"993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054\": container with ID starting with 993ecfe23bfad8bb2a1e1bf17cc631cbfccebc7cabb9b282a476960b20e89054 not found: ID does not exist" Nov 03 17:34:26.649250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649229 4980 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-installation-pull-secrets\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649251 4980 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-bound-sa-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649261 4980 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649270 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntgqq\" (UniqueName: \"kubernetes.io/projected/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-kube-api-access-ntgqq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649278 4980 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-ca-trust-extracted\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649305 4980 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-registry-certificates\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649314 4980 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-trusted-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.649436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.649323 4980 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e-image-registry-private-configuration\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:26.790248 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.790215 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:34:26.796869 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:26.796845 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6b4c9976fc-ckgdt"] Nov 03 17:34:28.291477 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:28.291445 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" path="/var/lib/kubelet/pods/5f42720f-0f0f-4b89-a6e3-6c4b0e42260e/volumes" Nov 03 17:34:44.518340 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:34:44.518304 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:34:49.092281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.092241 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerName="registry" containerID="cri-o://d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca" gracePeriod=30 Nov 03 17:34:49.338814 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.338784 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:34:49.420911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.420835 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.420911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.420876 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.420911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.420905 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.420962 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.420988 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.421012 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.421102 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.421150 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") pod \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\" (UID: \"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94\") " Nov 03 17:34:49.421504 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.421476 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:49.421818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.421786 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:34:49.423766 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.423726 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "kube-api-access-r54kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:49.423868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.423770 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:49.424022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.423981 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:49.424239 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.424090 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:34:49.424239 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.424204 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:34:49.429014 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.428991 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted" pod "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" (UID: "9d87a0d2-2d79-4cff-8a36-c8f3f6580b94"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:34:49.522290 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522261 4980 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-installation-pull-secrets\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522290 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522286 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r54kn\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-kube-api-access-r54kn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522298 4980 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-trusted-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522307 4980 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-certificates\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522316 4980 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-registry-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522324 4980 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-bound-sa-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522333 4980 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-image-registry-private-configuration\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.522450 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.522344 4980 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94-ca-trust-extracted\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:34:49.664539 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.664500 4980 generic.go:356] "Generic (PLEG): container finished" podID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerID="d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca" exitCode=0 Nov 03 17:34:49.664714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.664550 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" Nov 03 17:34:49.664714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.664603 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" event={"ID":"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94","Type":"ContainerDied","Data":"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca"} Nov 03 17:34:49.664714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.664656 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59985d89dd-5th7m" event={"ID":"9d87a0d2-2d79-4cff-8a36-c8f3f6580b94","Type":"ContainerDied","Data":"e47648405286276dafebf4f2889d57d2ccdbdaa76029a9c409ff89f59fb9a5cb"} Nov 03 17:34:49.664714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.664681 4980 scope.go:122] "RemoveContainer" containerID="d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca" Nov 03 17:34:49.673550 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.673534 4980 scope.go:122] "RemoveContainer" containerID="d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca" Nov 03 17:34:49.673837 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:34:49.673816 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca\": container with ID starting with d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca not found: ID does not exist" containerID="d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca" Nov 03 17:34:49.673899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.673846 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca"} err="failed to get container status \"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca\": rpc error: code = NotFound desc = could not find container \"d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca\": container with ID starting with d345a41c83feb5029ccd4924ef1a906e44ee9ff83039838a5f7bb0014eba4dca not found: ID does not exist" Nov 03 17:34:49.835386 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.835351 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:34:49.842522 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:49.842497 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-59985d89dd-5th7m"] Nov 03 17:34:50.291691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:50.291657 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" path="/var/lib/kubelet/pods/9d87a0d2-2d79-4cff-8a36-c8f3f6580b94/volumes" Nov 03 17:34:51.674409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:51.674374 4980 generic.go:356] "Generic (PLEG): container finished" podID="0a59ced0-24a7-45f0-a0c7-de168cbc59a7" containerID="892c11727184de612844e707852498c38ff5761c5e2198c6e2930072deebd5c0" exitCode=0 Nov 03 17:34:51.674791 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:51.674450 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" event={"ID":"0a59ced0-24a7-45f0-a0c7-de168cbc59a7","Type":"ContainerDied","Data":"892c11727184de612844e707852498c38ff5761c5e2198c6e2930072deebd5c0"} Nov 03 17:34:51.674874 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:51.674847 4980 scope.go:122] "RemoveContainer" containerID="892c11727184de612844e707852498c38ff5761c5e2198c6e2930072deebd5c0" Nov 03 17:34:52.682489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:34:52.682452 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-544d749c8c-jpwjc" event={"ID":"0a59ced0-24a7-45f0-a0c7-de168cbc59a7","Type":"ContainerStarted","Data":"651b6a0e8dc48f125ef13ba75399091a47e8b369ad3b8fa2e382c173509641bd"} Nov 03 17:35:05.729306 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:05.729272 4980 generic.go:356] "Generic (PLEG): container finished" podID="56cf9464-6022-41a2-b0bc-be87d6b3b90e" containerID="2494de23c64b3d041fe3eab953f56af62de082366fbd4861b6bd0800d9a351d9" exitCode=0 Nov 03 17:35:05.729857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:05.729352 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" event={"ID":"56cf9464-6022-41a2-b0bc-be87d6b3b90e","Type":"ContainerDied","Data":"2494de23c64b3d041fe3eab953f56af62de082366fbd4861b6bd0800d9a351d9"} Nov 03 17:35:05.729857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:05.729755 4980 scope.go:122] "RemoveContainer" containerID="2494de23c64b3d041fe3eab953f56af62de082366fbd4861b6bd0800d9a351d9" Nov 03 17:35:06.737724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:06.737685 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-7c4986d449-4x9kc" event={"ID":"56cf9464-6022-41a2-b0bc-be87d6b3b90e","Type":"ContainerStarted","Data":"afcee01d5f4a86b1c1a3d59ce6cd6ed89bcb777f6ea788ed3cf0c00ac0672894"} Nov 03 17:35:14.517296 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:35:14.517260 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:35:16.772794 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:16.772757 4980 generic.go:356] "Generic (PLEG): container finished" podID="9584d3a2-671c-48e3-89f9-dfd8e38db4ab" containerID="e7059d6ee070d5ae522c80169fcc30b8f9d2e82d503bf94118c968f497868341" exitCode=0 Nov 03 17:35:16.773303 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:16.772823 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" event={"ID":"9584d3a2-671c-48e3-89f9-dfd8e38db4ab","Type":"ContainerDied","Data":"e7059d6ee070d5ae522c80169fcc30b8f9d2e82d503bf94118c968f497868341"} Nov 03 17:35:16.773303 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:16.773198 4980 scope.go:122] "RemoveContainer" containerID="e7059d6ee070d5ae522c80169fcc30b8f9d2e82d503bf94118c968f497868341" Nov 03 17:35:17.778079 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:35:17.778045 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7d4bc45dd6-jdgpn" event={"ID":"9584d3a2-671c-48e3-89f9-dfd8e38db4ab","Type":"ContainerStarted","Data":"245aa1df6c02a129d6fe8c6cadc1bcc82785214ca8823d9a8c3557828b46487c"} Nov 03 17:35:44.518721 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:35:44.518681 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:36:14.517749 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:36:14.517672 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:36:44.518740 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:36:44.518701 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:37:14.518959 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:37:14.518918 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:37:44.518204 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:37:44.518169 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:38:14.519184 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:38:14.519142 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:38:40.307098 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:38:40.307072 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:38:40.307872 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:38:40.307854 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:38:40.312078 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:38:40.312061 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:38:40.312676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:38:40.312659 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:38:44.518599 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:38:44.518544 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:39:14.517916 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:39:14.517837 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:39:33.328416 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328386 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:39:33.328814 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328728 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" containerName="registry" Nov 03 17:39:33.328814 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328740 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" containerName="registry" Nov 03 17:39:33.328814 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328758 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerName="registry" Nov 03 17:39:33.328814 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328763 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerName="registry" Nov 03 17:39:33.328942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328815 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d87a0d2-2d79-4cff-8a36-c8f3f6580b94" containerName="registry" Nov 03 17:39:33.328942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.328828 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f42720f-0f0f-4b89-a6e3-6c4b0e42260e" containerName="registry" Nov 03 17:39:33.331687 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.331670 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.386742 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.386714 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Nov 03 17:39:33.386878 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.386813 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Nov 03 17:39:33.386878 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.386812 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Nov 03 17:39:33.386878 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.386846 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Nov 03 17:39:33.389755 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389738 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.389820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389764 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.389820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389782 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.389820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389800 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.389969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389826 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.389969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.389919 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8hpn\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.390078 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.390050 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.435275 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.435246 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-96fc4\"" Nov 03 17:39:33.436408 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.436390 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:39:33.491380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491347 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491381 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491638 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491399 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491638 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491418 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491638 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491436 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491638 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491597 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8hpn\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.491846 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.491672 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.492130 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.492099 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.493916 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.493892 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.494105 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.494084 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.494149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.494102 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.494188 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.494165 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.554525 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.554492 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.554695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.554679 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8hpn\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\") pod \"istiod-openshift-gateway-7876bfcb74-5ksj4\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:33.643359 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:33.643296 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:34.075410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:34.075370 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:39:34.078370 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:39:34.078343 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc712bc_4a2f_44c7_b60f_961228d0598a.slice/crio-35cfc3f7a6d32349e27de70b4dd934f88c2e089516118e7332aa0a094537aea6 WatchSource:0}: Error finding container 35cfc3f7a6d32349e27de70b4dd934f88c2e089516118e7332aa0a094537aea6: Status 404 returned error can't find the container with id 35cfc3f7a6d32349e27de70b4dd934f88c2e089516118e7332aa0a094537aea6 Nov 03 17:39:34.080286 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:34.080268 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:39:34.680633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:34.680592 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" event={"ID":"0fc712bc-4a2f-44c7-b60f-961228d0598a","Type":"ContainerStarted","Data":"35cfc3f7a6d32349e27de70b4dd934f88c2e089516118e7332aa0a094537aea6"} Nov 03 17:39:42.493758 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:42.493728 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:39:42.494016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:42.493792 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:39:42.714836 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:42.714795 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" event={"ID":"0fc712bc-4a2f-44c7-b60f-961228d0598a","Type":"ContainerStarted","Data":"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba"} Nov 03 17:39:42.715010 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:42.714889 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:39:42.839060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:42.838962 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" podStartSLOduration=1.4266209029999999 podStartE2EDuration="9.838948477s" totalImagesPullingTime="8.412327574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:39:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:39:42.837859406 +0000 UTC m=+362.666943011" watchObservedRunningTime="2026-11-03 17:39:42.838948477 +0000 UTC m=+362.668032079" Nov 03 17:39:43.719670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:43.719638 4980 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-5ksj4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:39:43.720046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:43.719686 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:39:44.519692 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:39:44.519653 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:39:46.719154 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:39:46.719124 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:40:14.518232 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:40:14.518197 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:40:23.763043 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.762952 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff"] Nov 03 17:40:23.766397 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.766379 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.878963 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.878921 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.879146 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.878969 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.879146 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.879072 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.879276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.879173 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gz5zk\" (UniqueName: \"kubernetes.io/projected/55694b88-f344-466c-9efa-a82edac4bdc6-kube-api-access-gz5zk\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.879276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.879208 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.879374 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.879296 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.893225 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893205 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Nov 03 17:40:23.893335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893255 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Nov 03 17:40:23.893335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893268 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Nov 03 17:40:23.893335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893324 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Nov 03 17:40:23.893462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893448 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Nov 03 17:40:23.893499 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.893478 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-kqpc6\"" Nov 03 17:40:23.980711 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980680 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.980711 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980716 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gz5zk\" (UniqueName: \"kubernetes.io/projected/55694b88-f344-466c-9efa-a82edac4bdc6-kube-api-access-gz5zk\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.980982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980735 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.980982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980782 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.980982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980827 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.980982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.980860 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.981196 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.981137 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-manifests\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.983237 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.983207 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/55694b88-f344-466c-9efa-a82edac4bdc6-tmp\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.983513 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.983495 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-metrics-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.983682 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.983661 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:23.983723 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:23.983661 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/55694b88-f344-466c-9efa-a82edac4bdc6-webhook-cert\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:24.037611 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:24.037515 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff"] Nov 03 17:40:24.044853 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:24.044827 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gz5zk\" (UniqueName: \"kubernetes.io/projected/55694b88-f344-466c-9efa-a82edac4bdc6-kube-api-access-gz5zk\") pod \"opendatahub-operator-controller-manager-57dbb86d66-6xwff\" (UID: \"55694b88-f344-466c-9efa-a82edac4bdc6\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:24.081779 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:24.081755 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:24.579048 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:24.579009 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff"] Nov 03 17:40:24.583991 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:40:24.583962 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55694b88_f344_466c_9efa_a82edac4bdc6.slice/crio-9e40a54c2e46cc8ca8484c40394074f8469676a94d5ef82395b0120ddbf4e649 WatchSource:0}: Error finding container 9e40a54c2e46cc8ca8484c40394074f8469676a94d5ef82395b0120ddbf4e649: Status 404 returned error can't find the container with id 9e40a54c2e46cc8ca8484c40394074f8469676a94d5ef82395b0120ddbf4e649 Nov 03 17:40:24.864586 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:24.864491 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" event={"ID":"55694b88-f344-466c-9efa-a82edac4bdc6","Type":"ContainerStarted","Data":"9e40a54c2e46cc8ca8484c40394074f8469676a94d5ef82395b0120ddbf4e649"} Nov 03 17:40:28.883297 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:28.883212 4980 generic.go:356] "Generic (PLEG): container finished" podID="55694b88-f344-466c-9efa-a82edac4bdc6" containerID="ae29478342d5f8e7959ff11e9b782aedcf5cfb35d8d64c96f0e38840d1d0fb37" exitCode=0 Nov 03 17:40:28.883719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:28.883286 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" event={"ID":"55694b88-f344-466c-9efa-a82edac4bdc6","Type":"ContainerDied","Data":"ae29478342d5f8e7959ff11e9b782aedcf5cfb35d8d64c96f0e38840d1d0fb37"} Nov 03 17:40:29.889237 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:29.889197 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" event={"ID":"55694b88-f344-466c-9efa-a82edac4bdc6","Type":"ContainerStarted","Data":"f4e7bb1a31295986d3c79918c629162f266bc165650ba78c641ed5dca50cac9b"} Nov 03 17:40:29.889664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:29.889310 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:30.135214 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:30.135168 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" podStartSLOduration=2.529518114 podStartE2EDuration="7.135153724s" totalImagesPullingTime="4.180937737s" totalInitContainerRuntime="424.697873ms" isStatefulPod=false podCreationTimestamp="2026-11-03 17:40:23 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:40:30.08512531 +0000 UTC m=+409.914208911" watchObservedRunningTime="2026-11-03 17:40:30.135153724 +0000 UTC m=+409.964237320" Nov 03 17:40:40.896206 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:40:40.896169 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-57dbb86d66-6xwff" Nov 03 17:40:44.518070 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:40:44.518028 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:41:03.966560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:03.966520 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r"] Nov 03 17:41:03.969487 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:03.969466 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.014507 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.014470 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.014676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.014514 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tmp\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.014676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.014609 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6qh5\" (UniqueName: \"kubernetes.io/projected/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-kube-api-access-z6qh5\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.025712 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.025692 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Nov 03 17:41:04.075677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.075635 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Nov 03 17:41:04.075839 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.075695 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-2zrqq\"" Nov 03 17:41:04.077040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.077019 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r"] Nov 03 17:41:04.115305 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.115279 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.115455 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.115311 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tmp\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.115455 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.115341 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6qh5\" (UniqueName: \"kubernetes.io/projected/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-kube-api-access-z6qh5\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.117844 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.117815 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tmp\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.118046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.118025 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-tls-certs\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.200000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.199971 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6qh5\" (UniqueName: \"kubernetes.io/projected/5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d-kube-api-access-z6qh5\") pod \"kube-auth-proxy-7c8f76b86f-nqp5r\" (UID: \"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d\") " pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.280276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.280197 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" Nov 03 17:41:04.670611 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:04.670512 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r"] Nov 03 17:41:04.674203 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:41:04.674163 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ff94d_8f0f_49d2_ab9c_93c646bc6f9d.slice/crio-9e18a8da0aa67c7cecb375a6ed59fd8229c6c4d6d244309dc2012cef4b4049ce WatchSource:0}: Error finding container 9e18a8da0aa67c7cecb375a6ed59fd8229c6c4d6d244309dc2012cef4b4049ce: Status 404 returned error can't find the container with id 9e18a8da0aa67c7cecb375a6ed59fd8229c6c4d6d244309dc2012cef4b4049ce Nov 03 17:41:05.032990 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:05.032955 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" event={"ID":"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d","Type":"ContainerStarted","Data":"9e18a8da0aa67c7cecb375a6ed59fd8229c6c4d6d244309dc2012cef4b4049ce"} Nov 03 17:41:08.047376 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.047334 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" event={"ID":"5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d","Type":"ContainerStarted","Data":"26265a192fd366dbafad0764cfbffc837cfcc094c4049e5e6366e1accb1aa853"} Nov 03 17:41:08.169532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.169475 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7c8f76b86f-nqp5r" podStartSLOduration=2.237608612 podStartE2EDuration="5.169458989s" totalImagesPullingTime="2.931850377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:08.168922713 +0000 UTC m=+447.998006315" watchObservedRunningTime="2026-11-03 17:41:08.169458989 +0000 UTC m=+447.998542594" Nov 03 17:41:08.226742 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.226708 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:08.234862 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.234839 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:08.248384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.248357 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgj7\" (UniqueName: \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\") pod \"authorino-57f7f8659f-r6glf\" (UID: \"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976\") " pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:08.289479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.289454 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 17:41:08.289691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.289545 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t8vd7\"" Nov 03 17:41:08.289868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.289552 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 17:41:08.293149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.293127 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:08.349324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.349256 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgj7\" (UniqueName: \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\") pod \"authorino-57f7f8659f-r6glf\" (UID: \"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976\") " pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:08.413597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.413543 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgj7\" (UniqueName: \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\") pod \"authorino-57f7f8659f-r6glf\" (UID: \"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976\") " pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:08.562961 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.562933 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:08.933841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:08.933748 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:08.936544 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:41:08.936512 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4bd6d8cb_fc2f_4f00_93e2_5fcd0db38976.slice/crio-179e6309aca33414b951637b681e5cae3468d786bc8625ad094d7111dbe3229d WatchSource:0}: Error finding container 179e6309aca33414b951637b681e5cae3468d786bc8625ad094d7111dbe3229d: Status 404 returned error can't find the container with id 179e6309aca33414b951637b681e5cae3468d786bc8625ad094d7111dbe3229d Nov 03 17:41:09.052505 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:09.052470 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r6glf" event={"ID":"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976","Type":"ContainerStarted","Data":"179e6309aca33414b951637b681e5cae3468d786bc8625ad094d7111dbe3229d"} Nov 03 17:41:14.519989 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:41:14.519955 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:41:15.081561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:15.081527 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r6glf" event={"ID":"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976","Type":"ContainerStarted","Data":"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672"} Nov 03 17:41:15.200700 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:15.200647 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-r6glf" podStartSLOduration=1.697515321 podStartE2EDuration="7.200632701s" totalImagesPullingTime="5.50311738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:15.198962925 +0000 UTC m=+455.028046527" watchObservedRunningTime="2026-11-03 17:41:15.200632701 +0000 UTC m=+455.029716302" Nov 03 17:41:15.564158 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:15.564121 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:17.088733 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:17.088683 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-r6glf" podUID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" containerName="authorino" containerID="cri-o://98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672" gracePeriod=30 Nov 03 17:41:17.329662 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:17.329636 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:17.432900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:17.432835 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\" (UniqueName: \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\") pod \"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976\" (UID: \"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976\") " Nov 03 17:41:17.435119 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:17.435092 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7" pod "4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" (UID: "4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976"). InnerVolumeSpecName "kube-api-access-lhgj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:41:17.533546 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:17.533513 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgj7\" (UniqueName: \"kubernetes.io/projected/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976-kube-api-access-lhgj7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:41:18.093925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.093886 4980 generic.go:356] "Generic (PLEG): container finished" podID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" containerID="98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672" exitCode=0 Nov 03 17:41:18.093925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.093927 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-r6glf" Nov 03 17:41:18.094433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.093928 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r6glf" event={"ID":"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976","Type":"ContainerDied","Data":"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672"} Nov 03 17:41:18.094433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.094047 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-r6glf" event={"ID":"4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976","Type":"ContainerDied","Data":"179e6309aca33414b951637b681e5cae3468d786bc8625ad094d7111dbe3229d"} Nov 03 17:41:18.094433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.094072 4980 scope.go:122] "RemoveContainer" containerID="98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672" Nov 03 17:41:18.103635 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.103618 4980 scope.go:122] "RemoveContainer" containerID="98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672" Nov 03 17:41:18.103877 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:41:18.103861 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672\": container with ID starting with 98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672 not found: ID does not exist" containerID="98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672" Nov 03 17:41:18.103923 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.103884 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672"} err="failed to get container status \"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672\": rpc error: code = NotFound desc = could not find container \"98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672\": container with ID starting with 98e41ba10a4826dea519d421289ae2405531dff9ca360206a029fb5ef5b64672 not found: ID does not exist" Nov 03 17:41:18.261047 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.261017 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:18.271730 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.271698 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-r6glf"] Nov 03 17:41:18.297102 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.297069 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" path="/var/lib/kubelet/pods/4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976/volumes" Nov 03 17:41:18.663114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.663080 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn"] Nov 03 17:41:18.663415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.663403 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" containerName="authorino" Nov 03 17:41:18.663467 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.663416 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" containerName="authorino" Nov 03 17:41:18.663505 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.663491 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bd6d8cb-fc2f-4f00-93e2-5fcd0db38976" containerName="authorino" Nov 03 17:41:18.667873 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.667854 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.722774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.722746 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Nov 03 17:41:18.744843 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.744812 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.744979 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.744853 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.744979 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.744952 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f75137-d10f-4d81-8c4f-5b8612ee4354-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.745096 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.744999 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gt5nd\" (UniqueName: \"kubernetes.io/projected/f7f75137-d10f-4d81-8c4f-5b8612ee4354-kube-api-access-gt5nd\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.771351 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.771318 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Nov 03 17:41:18.771528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.771407 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-module-controller-manager-dockercfg-cvzdt\"" Nov 03 17:41:18.846405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846367 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.846405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846411 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.846725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846437 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f75137-d10f-4d81-8c4f-5b8612ee4354-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.846725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846461 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gt5nd\" (UniqueName: \"kubernetes.io/projected/f7f75137-d10f-4d81-8c4f-5b8612ee4354-kube-api-access-gt5nd\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.846893 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846871 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-manifests\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.846960 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.846880 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f7f75137-d10f-4d81-8c4f-5b8612ee4354-tmp\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.872305 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.872269 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"odh-trusted-ca-bundle\"" Nov 03 17:41:18.873594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.873561 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn"] Nov 03 17:41:18.877397 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.877372 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f7f75137-d10f-4d81-8c4f-5b8612ee4354-trusted-ca\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.910069 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.910035 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gt5nd\" (UniqueName: \"kubernetes.io/projected/f7f75137-d10f-4d81-8c4f-5b8612ee4354-kube-api-access-gt5nd\") pod \"kserve-module-controller-manager-7566cf5fc8-ks4hn\" (UID: \"f7f75137-d10f-4d81-8c4f-5b8612ee4354\") " pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:18.981089 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:18.981057 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:19.408403 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:19.408317 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn"] Nov 03 17:41:19.411457 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:41:19.411425 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7f75137_d10f_4d81_8c4f_5b8612ee4354.slice/crio-035c3d3f3c971e78fdc4389986b179838b6bd1ca5a28a39f406ed303789f3240 WatchSource:0}: Error finding container 035c3d3f3c971e78fdc4389986b179838b6bd1ca5a28a39f406ed303789f3240: Status 404 returned error can't find the container with id 035c3d3f3c971e78fdc4389986b179838b6bd1ca5a28a39f406ed303789f3240 Nov 03 17:41:20.104183 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:20.104153 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" event={"ID":"f7f75137-d10f-4d81-8c4f-5b8612ee4354","Type":"ContainerStarted","Data":"035c3d3f3c971e78fdc4389986b179838b6bd1ca5a28a39f406ed303789f3240"} Nov 03 17:41:22.115100 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:22.115063 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" event={"ID":"f7f75137-d10f-4d81-8c4f-5b8612ee4354","Type":"ContainerStarted","Data":"94f8564c312b9c7fcfff41a37a87a01022cf7acd0be430069ba8cce8c316e8b4"} Nov 03 17:41:22.115516 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:22.115181 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:22.347222 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:22.347158 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" podStartSLOduration=2.454657682 podStartE2EDuration="4.347143108s" totalImagesPullingTime="1.892485426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:41:22.296159259 +0000 UTC m=+462.125242860" watchObservedRunningTime="2026-11-03 17:41:22.347143108 +0000 UTC m=+462.176226701" Nov 03 17:41:43.120695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:43.120663 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-module-controller-manager-7566cf5fc8-ks4hn" Nov 03 17:41:44.519691 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:41:44.519654 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:41:51.105330 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.105247 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6"] Nov 03 17:41:51.111422 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.111399 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.169212 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.169182 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6"] Nov 03 17:41:51.180471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.180449 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Nov 03 17:41:51.210197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.210170 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-data\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.210307 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.210211 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc5q\" (UniqueName: \"kubernetes.io/projected/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-kube-api-access-wtc5q\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.217270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.217244 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-tsd56\"" Nov 03 17:41:51.217369 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.217331 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Nov 03 17:41:51.217408 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.217391 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Nov 03 17:41:51.311282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.311239 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-data\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.311282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.311300 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc5q\" (UniqueName: \"kubernetes.io/projected/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-kube-api-access-wtc5q\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.311646 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.311627 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-data\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.375426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.375345 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc5q\" (UniqueName: \"kubernetes.io/projected/3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2-kube-api-access-wtc5q\") pod \"postgres-7cdbf6dc44-rk2t6\" (UID: \"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.423149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.423123 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:41:51.797182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:51.797149 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6"] Nov 03 17:41:51.799834 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:41:51.799802 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f2200c5_a9a7_4dfa_a15e_1779c6fa02f2.slice/crio-1246734617ff962919be3e791fc9f88b2ff0befe913ba08e39049086c4ea39a2 WatchSource:0}: Error finding container 1246734617ff962919be3e791fc9f88b2ff0befe913ba08e39049086c4ea39a2: Status 404 returned error can't find the container with id 1246734617ff962919be3e791fc9f88b2ff0befe913ba08e39049086c4ea39a2 Nov 03 17:41:52.235666 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:41:52.235628 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" event={"ID":"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2","Type":"ContainerStarted","Data":"1246734617ff962919be3e791fc9f88b2ff0befe913ba08e39049086c4ea39a2"} Nov 03 17:42:00.274752 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:00.274709 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" event={"ID":"3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2","Type":"ContainerStarted","Data":"3b7172fccaa8628c545cec06620231f4d553e646b30ee63d047e8615e13e8247"} Nov 03 17:42:00.275247 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:00.274859 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:42:00.394456 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:00.394388 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" podStartSLOduration=1.7750705930000001 podStartE2EDuration="9.394374024s" totalImagesPullingTime="7.619303431s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:00.393507279 +0000 UTC m=+500.222590883" watchObservedRunningTime="2026-11-03 17:42:00.394374024 +0000 UTC m=+500.223457626" Nov 03 17:42:06.308630 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:06.308592 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" Nov 03 17:42:14.090531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.090492 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:14.097001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.096970 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.152350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.152316 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bccd\"" Nov 03 17:42:14.152350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.152346 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 17:42:14.193909 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.193865 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwz6f\" (UniqueName: \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.194094 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.193934 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.194141 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.194112 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.200943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.200914 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 17:42:14.200943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.200929 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 17:42:14.202248 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.202224 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:14.295055 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.295016 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.295252 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.295062 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwz6f\" (UniqueName: \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.295252 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.295122 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.295252 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:14.295151 4980 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Nov 03 17:42:14.295252 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:14.295210 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert podName:b0d3811e-8fb2-40a9-a411-c222070cac71 nodeName:}" failed. No retries permitted until 2026-11-03 17:42:14.795195927 +0000 UTC m=+514.624279522 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert") pod "maas-controller-579cf7c6d-4hfm4" (UID: "b0d3811e-8fb2-40a9-a411-c222070cac71") : secret "maas-controller-webhook-cert" not found Nov 03 17:42:14.297811 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.297787 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.358619 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.358539 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwz6f\" (UniqueName: \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.518534 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:14.518487 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:42:14.531911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.531867 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 17:42:14.536350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.536332 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.598157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.598124 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.598157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.598161 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qs46\" (UniqueName: \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.598375 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.598196 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.600433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.600405 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 17:42:14.698701 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.698665 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.698701 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.698696 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qs46\" (UniqueName: \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.698955 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.698722 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.701340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.701316 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.701497 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.701359 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.762157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.762127 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qs46\" (UniqueName: \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\") pod \"maas-controller-b7df46f9c-kzlrh\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:14.799433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.799400 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.801890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.801863 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") pod \"maas-controller-579cf7c6d-4hfm4\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:14.849454 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:14.849419 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:15.009415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:15.009324 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:15.226713 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:15.226685 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 17:42:15.228933 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:15.228906 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28ca8e1d_78fd_4787_bb84_d3f57ffdce03.slice/crio-5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b WatchSource:0}: Error finding container 5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b: Status 404 returned error can't find the container with id 5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b Nov 03 17:42:15.335664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:15.335563 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" event={"ID":"28ca8e1d-78fd-4787-bb84-d3f57ffdce03","Type":"ContainerStarted","Data":"5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b"} Nov 03 17:42:15.391754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:15.391720 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:15.395074 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:15.395044 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0d3811e_8fb2_40a9_a411_c222070cac71.slice/crio-5776775b03fb8f07c46476b4ad1a2b9c7e3d1d35daabd9cdf4950a587fd19f0a WatchSource:0}: Error finding container 5776775b03fb8f07c46476b4ad1a2b9c7e3d1d35daabd9cdf4950a587fd19f0a: Status 404 returned error can't find the container with id 5776775b03fb8f07c46476b4ad1a2b9c7e3d1d35daabd9cdf4950a587fd19f0a Nov 03 17:42:16.342949 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:16.342878 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" event={"ID":"b0d3811e-8fb2-40a9-a411-c222070cac71","Type":"ContainerStarted","Data":"5776775b03fb8f07c46476b4ad1a2b9c7e3d1d35daabd9cdf4950a587fd19f0a"} Nov 03 17:42:19.359235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.359183 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" event={"ID":"28ca8e1d-78fd-4787-bb84-d3f57ffdce03","Type":"ContainerStarted","Data":"d3a15d1b175a3a7f9b0288f0755b73cf5fd2410255dac4dbeb763628d585d955"} Nov 03 17:42:19.359764 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.359263 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:19.360768 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.360747 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" event={"ID":"b0d3811e-8fb2-40a9-a411-c222070cac71","Type":"ContainerStarted","Data":"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4"} Nov 03 17:42:19.360903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.360862 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:19.483939 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.483882 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" podStartSLOduration=2.114778901 podStartE2EDuration="5.483866407s" totalImagesPullingTime="3.369087506s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:19.48311655 +0000 UTC m=+519.312200152" watchObservedRunningTime="2026-11-03 17:42:19.483866407 +0000 UTC m=+519.312950009" Nov 03 17:42:19.622060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:19.621966 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" podStartSLOduration=2.266194736 podStartE2EDuration="5.621952938s" totalImagesPullingTime="3.355758202s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:19.620629028 +0000 UTC m=+519.449712630" watchObservedRunningTime="2026-11-03 17:42:19.621952938 +0000 UTC m=+519.451036540" Nov 03 17:42:20.692562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.692529 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg"] Nov 03 17:42:20.702352 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.702323 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757015 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757172 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757053 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15395a8-9631-42fb-b9be-7eae057ebc92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757180 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757254 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757218 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757254 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757242 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757266 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.757350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.757304 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cr6j\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-kube-api-access-2cr6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.807196 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.807165 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg"] Nov 03 17:42:20.858723 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858688 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.858910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858737 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cr6j\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-kube-api-access-2cr6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.858910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858782 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.858910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858811 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15395a8-9631-42fb-b9be-7eae057ebc92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.858910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858870 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.858910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858905 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.859180 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.858935 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.859670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.859645 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.861337 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.861314 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.861410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.861315 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/e15395a8-9631-42fb-b9be-7eae057ebc92-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.861800 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.861773 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.861877 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.861782 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/e15395a8-9631-42fb-b9be-7eae057ebc92-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.930691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.930667 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:20.939985 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:20.939953 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cr6j\" (UniqueName: \"kubernetes.io/projected/e15395a8-9631-42fb-b9be-7eae057ebc92-kube-api-access-2cr6j\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7svwg\" (UID: \"e15395a8-9631-42fb-b9be-7eae057ebc92\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:21.012964 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:21.012872 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:21.445239 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:21.445210 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg"] Nov 03 17:42:21.447720 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:21.447686 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode15395a8_9631_42fb_b9be_7eae057ebc92.slice/crio-3a7a21a2e2ffcf8b104c435d31ed96799889e3a516a2cd7a5cd9b8fc1ae0159b WatchSource:0}: Error finding container 3a7a21a2e2ffcf8b104c435d31ed96799889e3a516a2cd7a5cd9b8fc1ae0159b: Status 404 returned error can't find the container with id 3a7a21a2e2ffcf8b104c435d31ed96799889e3a516a2cd7a5cd9b8fc1ae0159b Nov 03 17:42:21.449903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:21.449873 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:42:21.449998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:21.449934 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:42:22.376471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.376438 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" event={"ID":"e15395a8-9631-42fb-b9be-7eae057ebc92","Type":"ContainerStarted","Data":"2f3be4ca3f884865be77b281aabc4067447d5f208e653cab0e74d7959b142726"} Nov 03 17:42:22.376972 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.376489 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" event={"ID":"e15395a8-9631-42fb-b9be-7eae057ebc92","Type":"ContainerStarted","Data":"3a7a21a2e2ffcf8b104c435d31ed96799889e3a516a2cd7a5cd9b8fc1ae0159b"} Nov 03 17:42:22.376972 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.376610 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:22.378347 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.378323 4980 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7svwg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 17:42:22.378467 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.378365 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" podUID="e15395a8-9631-42fb-b9be-7eae057ebc92" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 17:42:22.586320 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:22.586252 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" podStartSLOduration=2.586231751 podStartE2EDuration="2.586231751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:22.549672061 +0000 UTC m=+522.378755662" watchObservedRunningTime="2026-11-03 17:42:22.586231751 +0000 UTC m=+522.415315354" Nov 03 17:42:23.381418 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.381390 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7svwg" Nov 03 17:42:23.606115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.606079 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:42:23.606374 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.606322 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerName="discovery" containerID="cri-o://438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba" gracePeriod=30 Nov 03 17:42:23.865770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.865736 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:42:23.989848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989620 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.989848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989691 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.989848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989746 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.989848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989797 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.989848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989830 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.990230 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989883 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.990230 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.989927 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\") pod \"0fc712bc-4a2f-44c7-b60f-961228d0598a\" (UID: \"0fc712bc-4a2f-44c7-b60f-961228d0598a\") " Nov 03 17:42:23.991682 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.991655 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:42:23.993859 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.993711 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:42:23.993859 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.993825 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "kube-api-access-b8hpn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:23.993859 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.993830 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:23.994093 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.993951 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:23.994728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.994707 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:23.995006 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:23.994976 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert" pod "0fc712bc-4a2f-44c7-b60f-961228d0598a" (UID: "0fc712bc-4a2f-44c7-b60f-961228d0598a"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:24.091004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.090974 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.090998 4980 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0fc712bc-4a2f-44c7-b60f-961228d0598a-local-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.091008 4980 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-dns-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.091019 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8hpn\" (UniqueName: \"kubernetes.io/projected/0fc712bc-4a2f-44c7-b60f-961228d0598a-kube-api-access-b8hpn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.091030 4980 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-cacerts\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.091038 4980 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-csr-ca-configmap\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.091217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.091046 4980 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0fc712bc-4a2f-44c7-b60f-961228d0598a-istio-kubeconfig\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:24.386137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.386102 4980 generic.go:356] "Generic (PLEG): container finished" podID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerID="438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba" exitCode=0 Nov 03 17:42:24.386562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.386161 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" Nov 03 17:42:24.386562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.386194 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" event={"ID":"0fc712bc-4a2f-44c7-b60f-961228d0598a","Type":"ContainerDied","Data":"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba"} Nov 03 17:42:24.386562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.386244 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4" event={"ID":"0fc712bc-4a2f-44c7-b60f-961228d0598a","Type":"ContainerDied","Data":"35cfc3f7a6d32349e27de70b4dd934f88c2e089516118e7332aa0a094537aea6"} Nov 03 17:42:24.386562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.386263 4980 scope.go:122] "RemoveContainer" containerID="438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba" Nov 03 17:42:24.394853 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.394838 4980 scope.go:122] "RemoveContainer" containerID="438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba" Nov 03 17:42:24.395109 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:24.395087 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba\": container with ID starting with 438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba not found: ID does not exist" containerID="438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba" Nov 03 17:42:24.395176 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.395121 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba"} err="failed to get container status \"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba\": rpc error: code = NotFound desc = could not find container \"438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba\": container with ID starting with 438468413aa9bfb7dd6834b7a88430a7153e6dfc60b0891e3a0a0df23c2cd1ba not found: ID does not exist" Nov 03 17:42:24.554181 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.554096 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:42:24.560081 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:24.560055 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-5ksj4"] Nov 03 17:42:26.292435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.292407 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" path="/var/lib/kubelet/pods/0fc712bc-4a2f-44c7-b60f-961228d0598a/volumes" Nov 03 17:42:26.469015 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.468983 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g98gb"] Nov 03 17:42:26.469330 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.469316 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerName="discovery" Nov 03 17:42:26.469379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.469332 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerName="discovery" Nov 03 17:42:26.469415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.469384 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fc712bc-4a2f-44c7-b60f-961228d0598a" containerName="discovery" Nov 03 17:42:26.473954 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.473930 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.576891 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.576821 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Nov 03 17:42:26.577053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.577033 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-5bzmz\"" Nov 03 17:42:26.578443 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.578424 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g98gb"] Nov 03 17:42:26.585983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.585962 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-9m7h2"] Nov 03 17:42:26.589241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.589224 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.614372 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.614350 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c649050-308d-47b6-81ae-5bcf17531e05-tls-certs\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.614468 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.614395 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8j8b\" (UniqueName: \"kubernetes.io/projected/3c649050-308d-47b6-81ae-5bcf17531e05-kube-api-access-k8j8b\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.642618 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.642595 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gxtcq\"" Nov 03 17:42:26.642768 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.642755 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Nov 03 17:42:26.642931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.642916 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Nov 03 17:42:26.650957 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.650935 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9m7h2"] Nov 03 17:42:26.715922 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.715895 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c649050-308d-47b6-81ae-5bcf17531e05-tls-certs\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.716057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.715941 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz55w\" (UniqueName: \"kubernetes.io/projected/27aefc84-7981-428b-a86a-69863cac4ec3-kube-api-access-hz55w\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.716057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.715975 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8j8b\" (UniqueName: \"kubernetes.io/projected/3c649050-308d-47b6-81ae-5bcf17531e05-kube-api-access-k8j8b\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.716057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.715998 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-cert\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.716057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.716020 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-metrics-certs\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.718358 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.718335 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c649050-308d-47b6-81ae-5bcf17531e05-tls-certs\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.778879 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.778851 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8j8b\" (UniqueName: \"kubernetes.io/projected/3c649050-308d-47b6-81ae-5bcf17531e05-kube-api-access-k8j8b\") pod \"model-serving-api-75bd89fdfc-g98gb\" (UID: \"3c649050-308d-47b6-81ae-5bcf17531e05\") " pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.786515 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.786494 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:26.816595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.816547 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz55w\" (UniqueName: \"kubernetes.io/projected/27aefc84-7981-428b-a86a-69863cac4ec3-kube-api-access-hz55w\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.816736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.816632 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-cert\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.816736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.816675 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-metrics-certs\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.819224 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.819197 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-metrics-certs\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.819342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.819317 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27aefc84-7981-428b-a86a-69863cac4ec3-cert\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.879905 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.879868 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz55w\" (UniqueName: \"kubernetes.io/projected/27aefc84-7981-428b-a86a-69863cac4ec3-kube-api-access-hz55w\") pod \"odh-model-controller-769875db56-9m7h2\" (UID: \"27aefc84-7981-428b-a86a-69863cac4ec3\") " pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:26.902698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:26.902672 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:27.162848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:27.162779 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-g98gb"] Nov 03 17:42:27.166308 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:27.166280 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c649050_308d_47b6_81ae_5bcf17531e05.slice/crio-53dce877b259f9554018f03fb9653f1a5d9f1ec424bc37ec7ffa23b50e41726d WatchSource:0}: Error finding container 53dce877b259f9554018f03fb9653f1a5d9f1ec424bc37ec7ffa23b50e41726d: Status 404 returned error can't find the container with id 53dce877b259f9554018f03fb9653f1a5d9f1ec424bc37ec7ffa23b50e41726d Nov 03 17:42:27.278599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:27.278558 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-9m7h2"] Nov 03 17:42:27.281072 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:27.281043 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27aefc84_7981_428b_a86a_69863cac4ec3.slice/crio-eb7e4ea8eb54c3314c497bba6ca1d75f9a6db2adcbbb2b190089791cfcc8ab24 WatchSource:0}: Error finding container eb7e4ea8eb54c3314c497bba6ca1d75f9a6db2adcbbb2b190089791cfcc8ab24: Status 404 returned error can't find the container with id eb7e4ea8eb54c3314c497bba6ca1d75f9a6db2adcbbb2b190089791cfcc8ab24 Nov 03 17:42:27.401005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:27.400968 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" event={"ID":"3c649050-308d-47b6-81ae-5bcf17531e05","Type":"ContainerStarted","Data":"53dce877b259f9554018f03fb9653f1a5d9f1ec424bc37ec7ffa23b50e41726d"} Nov 03 17:42:27.402115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:27.402092 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9m7h2" event={"ID":"27aefc84-7981-428b-a86a-69863cac4ec3","Type":"ContainerStarted","Data":"eb7e4ea8eb54c3314c497bba6ca1d75f9a6db2adcbbb2b190089791cfcc8ab24"} Nov 03 17:42:30.369202 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.369123 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 17:42:30.369700 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.369205 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:30.419130 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.419052 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" event={"ID":"3c649050-308d-47b6-81ae-5bcf17531e05","Type":"ContainerStarted","Data":"9f7d6586b006ad7e847adf2473d039e4e33d9a53ee63a8f012b8a08502b2f043"} Nov 03 17:42:30.419304 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.419196 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:30.542676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.542541 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:30.542855 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.542730 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" podUID="b0d3811e-8fb2-40a9-a411-c222070cac71" containerName="manager" containerID="cri-o://70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4" gracePeriod=30 Nov 03 17:42:30.813404 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.813375 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:30.856580 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.856514 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" podStartSLOduration=1.977037058 podStartE2EDuration="4.856499433s" totalImagesPullingTime="2.879462375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:30.734667051 +0000 UTC m=+530.563750653" watchObservedRunningTime="2026-11-03 17:42:30.856499433 +0000 UTC m=+530.685583034" Nov 03 17:42:30.952291 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.952262 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") pod \"b0d3811e-8fb2-40a9-a411-c222070cac71\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " Nov 03 17:42:30.952482 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.952342 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\" (UniqueName: \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\") pod \"b0d3811e-8fb2-40a9-a411-c222070cac71\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " Nov 03 17:42:30.952482 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.952417 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\") pod \"b0d3811e-8fb2-40a9-a411-c222070cac71\" (UID: \"b0d3811e-8fb2-40a9-a411-c222070cac71\") " Nov 03 17:42:30.954748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.954721 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert" pod "b0d3811e-8fb2-40a9-a411-c222070cac71" (UID: "b0d3811e-8fb2-40a9-a411-c222070cac71"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:30.954864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.954752 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs" pod "b0d3811e-8fb2-40a9-a411-c222070cac71" (UID: "b0d3811e-8fb2-40a9-a411-c222070cac71"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:42:30.954864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:30.954790 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f" pod "b0d3811e-8fb2-40a9-a411-c222070cac71" (UID: "b0d3811e-8fb2-40a9-a411-c222070cac71"). InnerVolumeSpecName "kube-api-access-vwz6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:42:31.053670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.053642 4980 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-webhook-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:31.053670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.053669 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwz6f\" (UniqueName: \"kubernetes.io/projected/b0d3811e-8fb2-40a9-a411-c222070cac71-kube-api-access-vwz6f\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:31.053862 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.053680 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0d3811e-8fb2-40a9-a411-c222070cac71-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:42:31.424606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.424490 4980 generic.go:356] "Generic (PLEG): container finished" podID="b0d3811e-8fb2-40a9-a411-c222070cac71" containerID="70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4" exitCode=0 Nov 03 17:42:31.424606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.424547 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" Nov 03 17:42:31.425127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.424603 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" event={"ID":"b0d3811e-8fb2-40a9-a411-c222070cac71","Type":"ContainerDied","Data":"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4"} Nov 03 17:42:31.425127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.424658 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-579cf7c6d-4hfm4" event={"ID":"b0d3811e-8fb2-40a9-a411-c222070cac71","Type":"ContainerDied","Data":"5776775b03fb8f07c46476b4ad1a2b9c7e3d1d35daabd9cdf4950a587fd19f0a"} Nov 03 17:42:31.425127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.424676 4980 scope.go:122] "RemoveContainer" containerID="70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4" Nov 03 17:42:31.426433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.426404 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-9m7h2" event={"ID":"27aefc84-7981-428b-a86a-69863cac4ec3","Type":"ContainerStarted","Data":"6855a4cd8a96ca37f69c05fc3781c03f1bb381cd2f56a7e571fe78eae80d8e0f"} Nov 03 17:42:31.426560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.426479 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:31.435440 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.435424 4980 scope.go:122] "RemoveContainer" containerID="70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4" Nov 03 17:42:31.435692 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:31.435674 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4\": container with ID starting with 70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4 not found: ID does not exist" containerID="70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4" Nov 03 17:42:31.435737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.435702 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4"} err="failed to get container status \"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4\": rpc error: code = NotFound desc = could not find container \"70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4\": container with ID starting with 70ffeecc93faa035d152ef2d7b58b37934fd22778d98c1bd39ca11d972b21cd4 not found: ID does not exist" Nov 03 17:42:31.598804 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.598752 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-9m7h2" podStartSLOduration=1.811535402 podStartE2EDuration="5.598737486s" totalImagesPullingTime="3.787202084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:31.597555275 +0000 UTC m=+531.426638888" watchObservedRunningTime="2026-11-03 17:42:31.598737486 +0000 UTC m=+531.427821130" Nov 03 17:42:31.713922 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.713890 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:31.717310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:31.717285 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-579cf7c6d-4hfm4"] Nov 03 17:42:32.292513 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:32.292482 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0d3811e-8fb2-40a9-a411-c222070cac71" path="/var/lib/kubelet/pods/b0d3811e-8fb2-40a9-a411-c222070cac71/volumes" Nov 03 17:42:41.431452 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:41.431422 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-g98gb" Nov 03 17:42:42.434559 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:42.434527 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-9m7h2" Nov 03 17:42:44.518214 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:42:44.518177 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:42:55.899316 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.899283 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:42:55.899723 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.899668 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b0d3811e-8fb2-40a9-a411-c222070cac71" containerName="manager" Nov 03 17:42:55.899723 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.899681 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0d3811e-8fb2-40a9-a411-c222070cac71" containerName="manager" Nov 03 17:42:55.899802 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.899755 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0d3811e-8fb2-40a9-a411-c222070cac71" containerName="manager" Nov 03 17:42:55.901548 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.901532 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:55.954268 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.954236 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:55.954445 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.954276 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrt8f\" (UniqueName: \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:55.954445 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.954393 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:55.954594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:55.954474 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.005287 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.005257 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fhmsl\"" Nov 03 17:42:56.005718 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.005702 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 17:42:56.005775 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.005762 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:42:56.007809 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.007788 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:42:56.055280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.055250 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.055422 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.055293 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrt8f\" (UniqueName: \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.055484 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.055423 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.055484 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.055471 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.056292 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.056251 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.057978 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.057947 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.058073 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.058027 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.118542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.118514 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrt8f\" (UniqueName: \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\") pod \"maas-api-6d669c7fcd-nmhfr\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.214236 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.214209 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:56.590294 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:56.590259 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:42:56.593142 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:42:56.593117 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb336f822_11ab_4423_a36e_084656166b21.slice/crio-47f27fd27b008ea8dd0055f550b12db0d94a62cc54ff899a02f2fda4be8d850d WatchSource:0}: Error finding container 47f27fd27b008ea8dd0055f550b12db0d94a62cc54ff899a02f2fda4be8d850d: Status 404 returned error can't find the container with id 47f27fd27b008ea8dd0055f550b12db0d94a62cc54ff899a02f2fda4be8d850d Nov 03 17:42:57.536409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:57.536371 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" event={"ID":"b336f822-11ab-4423-a36e-084656166b21","Type":"ContainerStarted","Data":"47f27fd27b008ea8dd0055f550b12db0d94a62cc54ff899a02f2fda4be8d850d"} Nov 03 17:42:59.546549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:59.546514 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" event={"ID":"b336f822-11ab-4423-a36e-084656166b21","Type":"ContainerStarted","Data":"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39"} Nov 03 17:42:59.546969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:59.546709 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:42:59.669241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:42:59.669186 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" podStartSLOduration=2.64764944 podStartE2EDuration="4.669172066s" totalImagesPullingTime="2.021522626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:42:59.668254807 +0000 UTC m=+559.497338449" watchObservedRunningTime="2026-11-03 17:42:59.669172066 +0000 UTC m=+559.498255668" Nov 03 17:43:05.555448 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:05.555420 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:43:06.872175 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.872138 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:06.876765 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.876743 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:06.931517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.931495 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 17:43:06.931657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.931598 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 17:43:06.937934 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.937909 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqzf5\" (UniqueName: \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\") pod \"authorino-5b9d7776c5-sp7d2\" (UID: \"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1\") " pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:06.980147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.980123 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t8vd7\"" Nov 03 17:43:06.981543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:06.981522 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:07.038497 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:07.038470 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqzf5\" (UniqueName: \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\") pod \"authorino-5b9d7776c5-sp7d2\" (UID: \"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1\") " pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:07.102730 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:07.102697 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqzf5\" (UniqueName: \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\") pod \"authorino-5b9d7776c5-sp7d2\" (UID: \"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1\") " pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:07.203770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:07.203741 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:07.282256 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:07.282225 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:07.576514 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:07.576448 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:07.579638 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:43:07.579607 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd4b22b8e_df09_4b40_bb89_2fa9a83ddba1.slice/crio-314d2bcf71815f4b3c621749f39c56c309ad02516b388909e2d2f7ee2065674e WatchSource:0}: Error finding container 314d2bcf71815f4b3c621749f39c56c309ad02516b388909e2d2f7ee2065674e: Status 404 returned error can't find the container with id 314d2bcf71815f4b3c621749f39c56c309ad02516b388909e2d2f7ee2065674e Nov 03 17:43:08.586511 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:08.586477 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" event={"ID":"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1","Type":"ContainerStarted","Data":"314d2bcf71815f4b3c621749f39c56c309ad02516b388909e2d2f7ee2065674e"} Nov 03 17:43:09.592618 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.592508 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" event={"ID":"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1","Type":"ContainerStarted","Data":"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446"} Nov 03 17:43:09.593108 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.592619 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" podUID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" containerName="authorino" containerID="cri-o://2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446" gracePeriod=30 Nov 03 17:43:09.711962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.711909 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" podStartSLOduration=2.051557131 podStartE2EDuration="3.711896593s" totalImagesPullingTime="1.660339462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:43:09.710704504 +0000 UTC m=+569.539788108" watchObservedRunningTime="2026-11-03 17:43:09.711896593 +0000 UTC m=+569.540980185" Nov 03 17:43:09.830977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.830949 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:09.865430 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.865371 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\" (UniqueName: \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\") pod \"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1\" (UID: \"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1\") " Nov 03 17:43:09.867540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.867515 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5" pod "d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" (UID: "d4b22b8e-df09-4b40-bb89-2fa9a83ddba1"). InnerVolumeSpecName "kube-api-access-qqzf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:09.966225 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:09.966191 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqzf5\" (UniqueName: \"kubernetes.io/projected/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1-kube-api-access-qqzf5\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:43:10.598062 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.597965 4980 generic.go:356] "Generic (PLEG): container finished" podID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" containerID="2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446" exitCode=0 Nov 03 17:43:10.598062 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.598008 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" Nov 03 17:43:10.598554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.598053 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" event={"ID":"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1","Type":"ContainerDied","Data":"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446"} Nov 03 17:43:10.598554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.598096 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5b9d7776c5-sp7d2" event={"ID":"d4b22b8e-df09-4b40-bb89-2fa9a83ddba1","Type":"ContainerDied","Data":"314d2bcf71815f4b3c621749f39c56c309ad02516b388909e2d2f7ee2065674e"} Nov 03 17:43:10.598554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.598112 4980 scope.go:122] "RemoveContainer" containerID="2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446" Nov 03 17:43:10.606372 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.606350 4980 scope.go:122] "RemoveContainer" containerID="2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446" Nov 03 17:43:10.606670 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:43:10.606636 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446\": container with ID starting with 2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446 not found: ID does not exist" containerID="2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446" Nov 03 17:43:10.606740 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.606682 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446"} err="failed to get container status \"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446\": rpc error: code = NotFound desc = could not find container \"2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446\": container with ID starting with 2388982a9bc7ecb2bb2f028202beb0a6c4c314bc18a95759258c91121adff446 not found: ID does not exist" Nov 03 17:43:10.765314 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.765282 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:10.815207 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:10.815179 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5b9d7776c5-sp7d2"] Nov 03 17:43:12.296444 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:12.293135 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" path="/var/lib/kubelet/pods/d4b22b8e-df09-4b40-bb89-2fa9a83ddba1/volumes" Nov 03 17:43:14.518652 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:43:14.518616 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:43:17.268233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.268197 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:43:17.269990 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.268404 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" podUID="b336f822-11ab-4423-a36e-084656166b21" containerName="maas-api" containerID="cri-o://8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39" gracePeriod=30 Nov 03 17:43:17.514987 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.514964 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:43:17.628981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.628893 4980 generic.go:356] "Generic (PLEG): container finished" podID="b336f822-11ab-4423-a36e-084656166b21" containerID="8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39" exitCode=0 Nov 03 17:43:17.628981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.628943 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" Nov 03 17:43:17.629176 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.628974 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" event={"ID":"b336f822-11ab-4423-a36e-084656166b21","Type":"ContainerDied","Data":"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39"} Nov 03 17:43:17.629176 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.629027 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr" event={"ID":"b336f822-11ab-4423-a36e-084656166b21","Type":"ContainerDied","Data":"47f27fd27b008ea8dd0055f550b12db0d94a62cc54ff899a02f2fda4be8d850d"} Nov 03 17:43:17.629176 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.629061 4980 scope.go:122] "RemoveContainer" containerID="8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39" Nov 03 17:43:17.632220 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.632199 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\" (UniqueName: \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\") pod \"b336f822-11ab-4423-a36e-084656166b21\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " Nov 03 17:43:17.632318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.632295 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\") pod \"b336f822-11ab-4423-a36e-084656166b21\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " Nov 03 17:43:17.632357 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.632334 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\") pod \"b336f822-11ab-4423-a36e-084656166b21\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " Nov 03 17:43:17.632414 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.632399 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\") pod \"b336f822-11ab-4423-a36e-084656166b21\" (UID: \"b336f822-11ab-4423-a36e-084656166b21\") " Nov 03 17:43:17.632854 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.632829 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca" pod "b336f822-11ab-4423-a36e-084656166b21" (UID: "b336f822-11ab-4423-a36e-084656166b21"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:43:17.634829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.634802 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f" pod "b336f822-11ab-4423-a36e-084656166b21" (UID: "b336f822-11ab-4423-a36e-084656166b21"). InnerVolumeSpecName "kube-api-access-zrt8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:43:17.635070 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.635040 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs" pod "b336f822-11ab-4423-a36e-084656166b21" (UID: "b336f822-11ab-4423-a36e-084656166b21"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:17.635160 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.635070 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls" pod "b336f822-11ab-4423-a36e-084656166b21" (UID: "b336f822-11ab-4423-a36e-084656166b21"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 17:43:17.639498 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.639282 4980 scope.go:122] "RemoveContainer" containerID="8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39" Nov 03 17:43:17.640039 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:43:17.640007 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39\": container with ID starting with 8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39 not found: ID does not exist" containerID="8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39" Nov 03 17:43:17.640158 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.640049 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39"} err="failed to get container status \"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39\": rpc error: code = NotFound desc = could not find container \"8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39\": container with ID starting with 8432b8a9a9b5a3b4320139c1f37d0e73645e64623dd82a8795f57d1cb92e4e39 not found: ID does not exist" Nov 03 17:43:17.733195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.733163 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b336f822-11ab-4423-a36e-084656166b21-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:43:17.733195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.733195 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrt8f\" (UniqueName: \"kubernetes.io/projected/b336f822-11ab-4423-a36e-084656166b21-kube-api-access-zrt8f\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:43:17.733389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.733208 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:43:17.733389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:17.733216 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b336f822-11ab-4423-a36e-084656166b21-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:43:18.053076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:18.053041 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:43:18.103108 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:18.103075 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-nmhfr"] Nov 03 17:43:18.292540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:18.292494 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b336f822-11ab-4423-a36e-084656166b21" path="/var/lib/kubelet/pods/b336f822-11ab-4423-a36e-084656166b21/volumes" Nov 03 17:43:40.336413 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:40.336373 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:43:40.338040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:40.338021 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:43:40.343058 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:40.343032 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:43:40.344072 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:43:40.344051 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:43:44.517934 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:43:44.517894 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:44:14.519366 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:44:14.519321 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:44:44.518281 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:44:44.518242 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:44:45.743265 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743234 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj"] Nov 03 17:44:45.743744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743678 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b336f822-11ab-4423-a36e-084656166b21" containerName="maas-api" Nov 03 17:44:45.743744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743691 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b336f822-11ab-4423-a36e-084656166b21" containerName="maas-api" Nov 03 17:44:45.743744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743701 4980 cpu_manager.go:414] "containerMap: removing container" podUID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" containerName="authorino" Nov 03 17:44:45.743744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743707 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" containerName="authorino" Nov 03 17:44:45.743875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743768 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b336f822-11ab-4423-a36e-084656166b21" containerName="maas-api" Nov 03 17:44:45.743875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.743776 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4b22b8e-df09-4b40-bb89-2fa9a83ddba1" containerName="authorino" Nov 03 17:44:45.746863 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.746840 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762109 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762080 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jxb4\" (UniqueName: \"kubernetes.io/projected/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-kube-api-access-2jxb4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762220 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762131 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762268 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762248 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762315 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762290 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762391 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762369 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.762503 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.762487 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.802712 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.802687 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-52z8v\"" Nov 03 17:44:45.852736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.852703 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Nov 03 17:44:45.852736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.852710 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Nov 03 17:44:45.852925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.852745 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Nov 03 17:44:45.854563 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.854534 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj"] Nov 03 17:44:45.863006 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.862974 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jxb4\" (UniqueName: \"kubernetes.io/projected/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-kube-api-access-2jxb4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863194 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863175 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863326 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863366 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863395 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863478 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.863921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863900 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.864011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.863899 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-home\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.864165 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.864140 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.866562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.866533 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.866562 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.866540 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-dshm\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:45.932745 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:45.932713 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jxb4\" (UniqueName: \"kubernetes.io/projected/bf5ad4c1-ae4c-487b-baa4-bf5385b1420e-kube-api-access-2jxb4\") pod \"premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj\" (UID: \"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e\") " pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:46.071912 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:46.071830 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:46.449704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:46.449656 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj"] Nov 03 17:44:46.452382 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:44:46.452354 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf5ad4c1_ae4c_487b_baa4_bf5385b1420e.slice/crio-3e46df3618abcd1bcf73fa9c1939bad49055ea34293cef53b3d51f1049e6ecf1 WatchSource:0}: Error finding container 3e46df3618abcd1bcf73fa9c1939bad49055ea34293cef53b3d51f1049e6ecf1: Status 404 returned error can't find the container with id 3e46df3618abcd1bcf73fa9c1939bad49055ea34293cef53b3d51f1049e6ecf1 Nov 03 17:44:46.453834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:46.453819 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:44:46.997633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:46.997600 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" event={"ID":"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e","Type":"ContainerStarted","Data":"3e46df3618abcd1bcf73fa9c1939bad49055ea34293cef53b3d51f1049e6ecf1"} Nov 03 17:44:49.008451 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.008419 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" event={"ID":"bf5ad4c1-ae4c-487b-baa4-bf5385b1420e","Type":"ContainerStarted","Data":"f9b8b3a0f120aedb24c1a601ca861b117e3f25f00a8d867866ffffd61925571f"} Nov 03 17:44:49.072435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.072401 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:49.137314 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.137265 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podStartSLOduration=1.9457870210000001 podStartE2EDuration="4.137252347s" totalImagesPullingTime="2.191465326s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:49.136173634 +0000 UTC m=+668.965257252" watchObservedRunningTime="2026-11-03 17:44:49.137252347 +0000 UTC m=+668.966335946" Nov 03 17:44:49.760994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.760958 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb"] Nov 03 17:44:49.764541 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.764513 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796620 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796807 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796658 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdqk4\" (UniqueName: \"kubernetes.io/projected/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-kube-api-access-jdqk4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796807 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796689 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796805 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796836 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.796897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.796860 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.868048 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.868019 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Nov 03 17:44:49.869379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.869357 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb"] Nov 03 17:44:49.898349 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898322 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898505 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898366 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898548 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898498 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdqk4\" (UniqueName: \"kubernetes.io/projected/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-kube-api-access-jdqk4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898645 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898542 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898645 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898637 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898781 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898672 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898786 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.898998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.898977 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.899065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.899051 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.900893 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.900868 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.901252 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.901226 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:49.963776 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:49.963742 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdqk4\" (UniqueName: \"kubernetes.io/projected/5172ab4a-c7f4-4fac-a164-b77afacc7c5c-kube-api-access-jdqk4\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb\" (UID: \"5172ab4a-c7f4-4fac-a164-b77afacc7c5c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:50.077018 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:50.076939 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:50.456204 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:50.456107 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb"] Nov 03 17:44:50.458846 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:44:50.458816 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5172ab4a_c7f4_4fac_a164_b77afacc7c5c.slice/crio-6c9df561d0ab18a4565c528f9e1a810c63ff7fea9d8d60339120f675cf4eff34 WatchSource:0}: Error finding container 6c9df561d0ab18a4565c528f9e1a810c63ff7fea9d8d60339120f675cf4eff34: Status 404 returned error can't find the container with id 6c9df561d0ab18a4565c528f9e1a810c63ff7fea9d8d60339120f675cf4eff34 Nov 03 17:44:51.017922 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:51.017879 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" event={"ID":"5172ab4a-c7f4-4fac-a164-b77afacc7c5c","Type":"ContainerStarted","Data":"6c9df561d0ab18a4565c528f9e1a810c63ff7fea9d8d60339120f675cf4eff34"} Nov 03 17:44:52.023675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:52.023642 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" event={"ID":"5172ab4a-c7f4-4fac-a164-b77afacc7c5c","Type":"ContainerStarted","Data":"7af4e7cd75bf8b7b5ee6b846d5da553514f492fe80386d3f185e407ce2e72768"} Nov 03 17:44:52.077057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:52.077028 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:44:52.147012 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:52.146962 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" podStartSLOduration=2.43858833 podStartE2EDuration="3.146947215s" totalImagesPullingTime="708.358885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:52.146165198 +0000 UTC m=+671.975248800" watchObservedRunningTime="2026-11-03 17:44:52.146947215 +0000 UTC m=+671.976030817" Nov 03 17:44:53.871036 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.871002 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j7n9l"] Nov 03 17:44:53.874449 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.874431 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935434 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935393 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935635 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935554 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a742aa19-68c7-4725-8b5c-0c988255eb17-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935794 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935773 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935811 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-home\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935838 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhbjp\" (UniqueName: \"kubernetes.io/projected/a742aa19-68c7-4725-8b5c-0c988255eb17-kube-api-access-lhbjp\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.935900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.935878 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:53.977804 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.977777 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Nov 03 17:44:53.979616 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:53.979589 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j7n9l"] Nov 03 17:44:54.036917 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.036876 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.036917 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.036921 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a742aa19-68c7-4725-8b5c-0c988255eb17-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.036968 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.036987 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-home\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.037004 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhbjp\" (UniqueName: \"kubernetes.io/projected/a742aa19-68c7-4725-8b5c-0c988255eb17-kube-api-access-lhbjp\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.037029 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.037334 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.037350 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.037478 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.037425 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-home\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.039382 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.039359 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a742aa19-68c7-4725-8b5c-0c988255eb17-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.039691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.039671 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a742aa19-68c7-4725-8b5c-0c988255eb17-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.100447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.100417 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhbjp\" (UniqueName: \"kubernetes.io/projected/a742aa19-68c7-4725-8b5c-0c988255eb17-kube-api-access-lhbjp\") pod \"echo-simulator-kserve-64b6f8f96c-j7n9l\" (UID: \"a742aa19-68c7-4725-8b5c-0c988255eb17\") " pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.187185 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.187114 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:54.564708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:54.564682 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-j7n9l"] Nov 03 17:44:54.567103 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:44:54.567075 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda742aa19_68c7_4725_8b5c_0c988255eb17.slice/crio-e5b0e2e13efe85652f4d1b8b493e46a7cb76554f212378a028ae87f164af3095 WatchSource:0}: Error finding container e5b0e2e13efe85652f4d1b8b493e46a7cb76554f212378a028ae87f164af3095: Status 404 returned error can't find the container with id e5b0e2e13efe85652f4d1b8b493e46a7cb76554f212378a028ae87f164af3095 Nov 03 17:44:55.036911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:55.036880 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" event={"ID":"a742aa19-68c7-4725-8b5c-0c988255eb17","Type":"ContainerStarted","Data":"e5b0e2e13efe85652f4d1b8b493e46a7cb76554f212378a028ae87f164af3095"} Nov 03 17:44:56.072920 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:56.072878 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:56.087026 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:56.086994 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:57.048518 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:57.048486 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" event={"ID":"a742aa19-68c7-4725-8b5c-0c988255eb17","Type":"ContainerStarted","Data":"ed93a3cbb270263d8ab3e4453f956ddb575f779af3254328faa078b753248bc2"} Nov 03 17:44:57.060303 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:57.060275 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:44:57.170556 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:57.170450 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" podStartSLOduration=1.776757804 podStartE2EDuration="4.17043241s" totalImagesPullingTime="2.393674606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:44:57.170160928 +0000 UTC m=+676.999244600" watchObservedRunningTime="2026-11-03 17:44:57.17043241 +0000 UTC m=+676.999516013" Nov 03 17:44:57.188089 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:57.188065 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:44:58.069888 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.069854 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4"] Nov 03 17:44:58.073247 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.073229 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.127870 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.127847 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Nov 03 17:44:58.175561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.175528 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpx9z\" (UniqueName: \"kubernetes.io/projected/07ed2d59-e332-4426-bc90-87189e39ef2e-kube-api-access-wpx9z\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.176000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.175609 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.176000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.175726 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ed2d59-e332-4426-bc90-87189e39ef2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.176000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.175896 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.176000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.175966 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.176223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.176050 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.178319 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.178298 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4"] Nov 03 17:44:58.276723 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276693 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpx9z\" (UniqueName: \"kubernetes.io/projected/07ed2d59-e332-4426-bc90-87189e39ef2e-kube-api-access-wpx9z\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.276899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276738 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.276899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276778 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ed2d59-e332-4426-bc90-87189e39ef2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.276899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276840 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.276899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276879 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.277119 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.276926 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.277223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.277191 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.277339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.277249 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.277339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.277288 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.279317 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.279299 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/07ed2d59-e332-4426-bc90-87189e39ef2e-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.279554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.279539 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/07ed2d59-e332-4426-bc90-87189e39ef2e-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.350402 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.350341 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpx9z\" (UniqueName: \"kubernetes.io/projected/07ed2d59-e332-4426-bc90-87189e39ef2e-kube-api-access-wpx9z\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4\" (UID: \"07ed2d59-e332-4426-bc90-87189e39ef2e\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.385948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.385917 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:44:58.762837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:58.762811 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4"] Nov 03 17:44:58.764861 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:44:58.764836 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07ed2d59_e332_4426_bc90_87189e39ef2e.slice/crio-8a1b978159d0215bab0674ef2f364e3e2f2a0e7b0106a99f89cb4a16d1e1b8f3 WatchSource:0}: Error finding container 8a1b978159d0215bab0674ef2f364e3e2f2a0e7b0106a99f89cb4a16d1e1b8f3: Status 404 returned error can't find the container with id 8a1b978159d0215bab0674ef2f364e3e2f2a0e7b0106a99f89cb4a16d1e1b8f3 Nov 03 17:44:59.058690 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:44:59.058594 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" event={"ID":"07ed2d59-e332-4426-bc90-87189e39ef2e","Type":"ContainerStarted","Data":"8a1b978159d0215bab0674ef2f364e3e2f2a0e7b0106a99f89cb4a16d1e1b8f3"} Nov 03 17:45:00.064471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.064430 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" event={"ID":"07ed2d59-e332-4426-bc90-87189e39ef2e","Type":"ContainerStarted","Data":"9f7ba1bdf9714cf956956a1eecc41d4af69eeba89f0b14eee7812fc0f82ecce6"} Nov 03 17:45:00.077137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.077108 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:45:00.089435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.089416 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:45:00.187632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.187566 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" podStartSLOduration=1.451420221 podStartE2EDuration="2.187551663s" totalImagesPullingTime="736.131442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:44:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:00.186529777 +0000 UTC m=+680.015613379" watchObservedRunningTime="2026-11-03 17:45:00.187551663 +0000 UTC m=+680.016635265" Nov 03 17:45:00.190256 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.190234 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:00.193681 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.193661 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.248951 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.248930 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 17:45:00.293235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.293205 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.293338 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.293247 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccz5r\" (UniqueName: \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.297943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.297914 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:00.386606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.386520 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:45:00.393913 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.393894 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.394003 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.393933 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccz5r\" (UniqueName: \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.394481 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.394462 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.457278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.457245 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccz5r\" (UniqueName: \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\") pod \"maas-api-key-cleanup-29895465-hw9q4\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.506285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.506260 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:00.883253 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:00.883210 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:00.886598 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:45:00.886536 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a413925_b5dc_4729_aba8_a7c8b7084b27.slice/crio-002b6d2ce386244d26d908f64cdb2bf54244f7717e1783490b0b0ecfa23d3da0 WatchSource:0}: Error finding container 002b6d2ce386244d26d908f64cdb2bf54244f7717e1783490b0b0ecfa23d3da0: Status 404 returned error can't find the container with id 002b6d2ce386244d26d908f64cdb2bf54244f7717e1783490b0b0ecfa23d3da0 Nov 03 17:45:01.069894 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:01.069856 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerStarted","Data":"002b6d2ce386244d26d908f64cdb2bf54244f7717e1783490b0b0ecfa23d3da0"} Nov 03 17:45:01.081233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:01.081207 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb" Nov 03 17:45:04.187785 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:04.187739 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:45:04.207608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:04.207551 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:45:05.097017 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:05.096976 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerStarted","Data":"e2796a15c2725d80a5fe530fdbf1aa1afd9e3c5ffd194c1e0e432a251c30f080"} Nov 03 17:45:05.108420 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:05.108397 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-j7n9l" Nov 03 17:45:05.218210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:05.218164 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" podStartSLOduration=1.483915313 podStartE2EDuration="5.218150882s" totalImagesPullingTime="3.734235569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:05.216793477 +0000 UTC m=+685.045877079" watchObservedRunningTime="2026-11-03 17:45:05.218150882 +0000 UTC m=+685.047234483" Nov 03 17:45:06.103265 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:06.103230 4980 generic.go:356] "Generic (PLEG): container finished" podID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerID="e2796a15c2725d80a5fe530fdbf1aa1afd9e3c5ffd194c1e0e432a251c30f080" exitCode=60 Nov 03 17:45:06.103446 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:06.103326 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerDied","Data":"e2796a15c2725d80a5fe530fdbf1aa1afd9e3c5ffd194c1e0e432a251c30f080"} Nov 03 17:45:06.103582 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:06.103549 4980 scope.go:122] "RemoveContainer" containerID="e2796a15c2725d80a5fe530fdbf1aa1afd9e3c5ffd194c1e0e432a251c30f080" Nov 03 17:45:07.110344 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:07.110260 4980 generic.go:356] "Generic (PLEG): container finished" podID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerID="06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33" exitCode=60 Nov 03 17:45:07.110820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:07.110362 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerDied","Data":"06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33"} Nov 03 17:45:07.110820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:07.110428 4980 scope.go:122] "RemoveContainer" containerID="e2796a15c2725d80a5fe530fdbf1aa1afd9e3c5ffd194c1e0e432a251c30f080" Nov 03 17:45:07.110820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:07.110694 4980 scope.go:122] "RemoveContainer" containerID="06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33" Nov 03 17:45:07.110988 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:45:07.110925 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-hw9q4_odh-ai-gateway-infra(3a413925-b5dc-4729-aba8-a7c8b7084b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" Nov 03 17:45:08.116313 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:08.116288 4980 scope.go:122] "RemoveContainer" containerID="06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33" Nov 03 17:45:08.116743 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:45:08.116478 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-hw9q4_odh-ai-gateway-infra(3a413925-b5dc-4729-aba8-a7c8b7084b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" Nov 03 17:45:08.386255 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:08.386169 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:45:08.398725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:08.398702 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:45:09.132436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:09.132396 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4" Nov 03 17:45:14.518690 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:45:14.518651 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:45:17.382549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.382514 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb"] Nov 03 17:45:17.389883 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.389858 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.447097 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.447066 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Nov 03 17:45:17.496816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.496782 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb"] Nov 03 17:45:17.535042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535011 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.535194 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535059 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnk5d\" (UniqueName: \"kubernetes.io/projected/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-kube-api-access-lnk5d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.535194 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535156 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.535270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535219 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.535270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535246 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.535331 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.535284 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636467 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636510 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636533 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636845 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636605 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636845 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636662 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.636946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.636837 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnk5d\" (UniqueName: \"kubernetes.io/projected/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-kube-api-access-lnk5d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.637023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.637002 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.637089 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.637034 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.637144 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.637116 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.639061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.639036 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.639544 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.639518 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:17.704592 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:17.704553 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnk5d\" (UniqueName: \"kubernetes.io/projected/00efe73b-9c70-4536-9b6f-cbd8e4ca188d-kube-api-access-lnk5d\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-f46zb\" (UID: \"00efe73b-9c70-4536-9b6f-cbd8e4ca188d\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:18.003071 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.003029 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:18.227807 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.227777 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:45:18.232473 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.232445 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.287995 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.287917 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\"" Nov 03 17:45:18.331968 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.331935 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:45:18.343204 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343170 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343235 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzkdm\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343388 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343338 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343388 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343371 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343407 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343502 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343480 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343534 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343511 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343534 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343530 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.343637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.343549 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.397639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.397540 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb"] Nov 03 17:45:18.400246 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:45:18.400222 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00efe73b_9c70_4536_9b6f_cbd8e4ca188d.slice/crio-c758459f6b28201a66e32b70bde2fa83b9aa8f255a5cebb63066c68c5ca3315b WatchSource:0}: Error finding container c758459f6b28201a66e32b70bde2fa83b9aa8f255a5cebb63066c68c5ca3315b: Status 404 returned error can't find the container with id c758459f6b28201a66e32b70bde2fa83b9aa8f255a5cebb63066c68c5ca3315b Nov 03 17:45:18.444559 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444527 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444594 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzkdm\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444638 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444666 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444701 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444722 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444988 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444794 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444988 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444839 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.444988 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.444864 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.445151 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.445061 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.445151 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.445119 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.445243 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.445159 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.445243 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.445195 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.445349 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.445331 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.447023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.447005 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.447334 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.447316 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.510074 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.510036 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.510228 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.510121 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzkdm\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-zgg6k\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.547078 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.547007 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:18.978993 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:18.978966 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:45:18.980937 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:45:18.980886 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod134977fe_81ee_45dc_8d23_b76703e17683.slice/crio-4d57330a85ca2816a1466948b1af04d94d1373d782ddd0e16f1f5a39345abe80 WatchSource:0}: Error finding container 4d57330a85ca2816a1466948b1af04d94d1373d782ddd0e16f1f5a39345abe80: Status 404 returned error can't find the container with id 4d57330a85ca2816a1466948b1af04d94d1373d782ddd0e16f1f5a39345abe80 Nov 03 17:45:19.166017 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:19.165978 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" event={"ID":"00efe73b-9c70-4536-9b6f-cbd8e4ca188d","Type":"ContainerStarted","Data":"c758459f6b28201a66e32b70bde2fa83b9aa8f255a5cebb63066c68c5ca3315b"} Nov 03 17:45:19.167715 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:19.167679 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" event={"ID":"134977fe-81ee-45dc-8d23-b76703e17683","Type":"ContainerStarted","Data":"4d57330a85ca2816a1466948b1af04d94d1373d782ddd0e16f1f5a39345abe80"} Nov 03 17:45:20.173375 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:20.173342 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" event={"ID":"00efe73b-9c70-4536-9b6f-cbd8e4ca188d","Type":"ContainerStarted","Data":"dcfe0743e3604fe79cb0c5007b1478a1a7220aa99031e8ad3c7c491278960be1"} Nov 03 17:45:20.286379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:20.286348 4980 scope.go:122] "RemoveContainer" containerID="06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33" Nov 03 17:45:20.310026 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:20.309981 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" podStartSLOduration=2.51448809 podStartE2EDuration="3.309968474s" totalImagesPullingTime="795.480384ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:20.300867169 +0000 UTC m=+700.129950767" watchObservedRunningTime="2026-11-03 17:45:20.309968474 +0000 UTC m=+700.139052077" Nov 03 17:45:21.003367 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:21.003332 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:21.179699 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:21.179666 4980 generic.go:356] "Generic (PLEG): container finished" podID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerID="9715bc99a7f44a3dcfdf8fc75fcc5235a308dfd1a663810418dff4d5cc76bc09" exitCode=60 Nov 03 17:45:21.180096 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:21.179740 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerDied","Data":"9715bc99a7f44a3dcfdf8fc75fcc5235a308dfd1a663810418dff4d5cc76bc09"} Nov 03 17:45:21.180096 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:21.179829 4980 scope.go:122] "RemoveContainer" containerID="06788dce045b5db8638a431d4b95c530636776efe8d20527b16580381a31de33" Nov 03 17:45:21.180198 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:21.180151 4980 scope.go:122] "RemoveContainer" containerID="9715bc99a7f44a3dcfdf8fc75fcc5235a308dfd1a663810418dff4d5cc76bc09" Nov 03 17:45:21.180349 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:45:21.180328 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895465-hw9q4_odh-ai-gateway-infra(3a413925-b5dc-4729-aba8-a7c8b7084b27)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" Nov 03 17:45:22.324133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.324090 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:22.483905 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.483877 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:22.581993 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.581909 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\") pod \"3a413925-b5dc-4729-aba8-a7c8b7084b27\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " Nov 03 17:45:22.582168 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.582020 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\" (UniqueName: \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\") pod \"3a413925-b5dc-4729-aba8-a7c8b7084b27\" (UID: \"3a413925-b5dc-4729-aba8-a7c8b7084b27\") " Nov 03 17:45:22.582351 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.582308 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca" pod "3a413925-b5dc-4729-aba8-a7c8b7084b27" (UID: "3a413925-b5dc-4729-aba8-a7c8b7084b27"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:45:22.584934 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.584905 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r" pod "3a413925-b5dc-4729-aba8-a7c8b7084b27" (UID: "3a413925-b5dc-4729-aba8-a7c8b7084b27"). InnerVolumeSpecName "kube-api-access-ccz5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:45:22.683031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.682999 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccz5r\" (UniqueName: \"kubernetes.io/projected/3a413925-b5dc-4729-aba8-a7c8b7084b27-kube-api-access-ccz5r\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:45:22.683031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:22.683029 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a413925-b5dc-4729-aba8-a7c8b7084b27-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:45:23.194039 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.194005 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" event={"ID":"3a413925-b5dc-4729-aba8-a7c8b7084b27","Type":"ContainerDied","Data":"002b6d2ce386244d26d908f64cdb2bf54244f7717e1783490b0b0ecfa23d3da0"} Nov 03 17:45:23.194039 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.194027 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4" Nov 03 17:45:23.194354 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.194097 4980 scope.go:122] "RemoveContainer" containerID="9715bc99a7f44a3dcfdf8fc75fcc5235a308dfd1a663810418dff4d5cc76bc09" Nov 03 17:45:23.332740 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.332705 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:23.333015 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.332789 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:23.333015 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.332841 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:23.369252 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.369207 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:23.392330 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:23.392305 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895465-hw9q4"] Nov 03 17:45:24.201516 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.201479 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" event={"ID":"134977fe-81ee-45dc-8d23-b76703e17683","Type":"ContainerStarted","Data":"6f8a54bd293fb7cc9617b34ab93ca0bdfbfcb52fe6ae24aabae140df6ee9bda9"} Nov 03 17:45:24.291832 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.291797 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" path="/var/lib/kubelet/pods/3a413925-b5dc-4729-aba8-a7c8b7084b27/volumes" Nov 03 17:45:24.383350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.383281 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podStartSLOduration=2.034640391 podStartE2EDuration="6.383261178s" totalImagesPullingTime="4.348620787s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:24.381358467 +0000 UTC m=+704.210442069" watchObservedRunningTime="2026-11-03 17:45:24.383261178 +0000 UTC m=+704.212344781" Nov 03 17:45:24.547754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.547677 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:24.548909 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.548883 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:24.549028 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:24.548931 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:25.547377 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:25.547346 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:25.547834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:25.547398 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:26.548270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:26.548228 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:26.548692 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:26.548295 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:27.547670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:27.547640 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:27.547850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:27.547686 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:28.004166 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:28.004128 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:28.016699 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:28.016675 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:28.230948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:28.230919 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" Nov 03 17:45:28.547971 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:28.547937 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:28.548168 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:28.547994 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:29.547525 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:29.547488 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:29.548010 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:29.547550 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:30.548064 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:30.548034 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" start-of-body= Nov 03 17:45:30.548472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:30.548084 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": dial tcp 10.132.0.44:15021: connect: connection refused" Nov 03 17:45:32.548874 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:32.548825 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:45:32.549409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:32.548886 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:45:33.547368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:33.547337 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:33.590750 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:33.590707 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-zgg6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:45:33.591178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:33.590767 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:45:34.223921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:34.223887 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:34.247490 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:34.247463 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:45:41.781666 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.781627 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-cbc5b6946-5pd45"] Nov 03 17:45:41.782059 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.781993 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.782059 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.782003 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.782059 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.782024 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.782059 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.782029 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.782192 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.782094 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.782192 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.782106 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:41.785364 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.785346 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:41.839841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.839808 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Nov 03 17:45:41.839841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.839819 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Nov 03 17:45:41.850518 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.850491 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-tls-cert\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:41.850664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.850549 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjs6k\" (UniqueName: \"kubernetes.io/projected/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-kube-api-access-pjs6k\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:41.888554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.888525 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t8vd7\"" Nov 03 17:45:41.888701 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.888609 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Nov 03 17:45:41.892024 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.892004 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbc5b6946-5pd45"] Nov 03 17:45:41.952155 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.952127 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-tls-cert\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:41.952300 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.952193 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjs6k\" (UniqueName: \"kubernetes.io/projected/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-kube-api-access-pjs6k\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:41.954849 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:41.954821 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-tls-cert\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:42.018914 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:42.018882 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjs6k\" (UniqueName: \"kubernetes.io/projected/d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d-kube-api-access-pjs6k\") pod \"authorino-cbc5b6946-5pd45\" (UID: \"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d\") " pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:42.096423 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:42.096350 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-cbc5b6946-5pd45" Nov 03 17:45:42.476219 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:42.476190 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-cbc5b6946-5pd45"] Nov 03 17:45:42.478045 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:45:42.478018 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5da6d0f_f0f5_49f2_a8ea_f4b2fa01291d.slice/crio-c1ac98a0c1fbfc16c16f420dce62767938c6da28e6ce8831baa8fb99a7a07dd7 WatchSource:0}: Error finding container c1ac98a0c1fbfc16c16f420dce62767938c6da28e6ce8831baa8fb99a7a07dd7: Status 404 returned error can't find the container with id c1ac98a0c1fbfc16c16f420dce62767938c6da28e6ce8831baa8fb99a7a07dd7 Nov 03 17:45:43.285522 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:43.285487 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc5b6946-5pd45" event={"ID":"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d","Type":"ContainerStarted","Data":"c1ac98a0c1fbfc16c16f420dce62767938c6da28e6ce8831baa8fb99a7a07dd7"} Nov 03 17:45:44.291954 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:44.291868 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-cbc5b6946-5pd45" event={"ID":"d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d","Type":"ContainerStarted","Data":"6bb7626750f28d83486fd869ba3c92c75f59351ca6981124eb4e7f8767a54ba1"} Nov 03 17:45:44.411172 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:44.411114 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-cbc5b6946-5pd45" podStartSLOduration=1.927449235 podStartE2EDuration="3.411098118s" totalImagesPullingTime="1.483648883s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:44.410967481 +0000 UTC m=+724.240051083" watchObservedRunningTime="2026-11-03 17:45:44.411098118 +0000 UTC m=+724.240181722" Nov 03 17:45:44.519598 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:45:44.519546 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:45:48.542022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.541989 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:45:48.542471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.542405 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:48.542471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.542419 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:48.542553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.542518 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a413925-b5dc-4729-aba8-a7c8b7084b27" containerName="cleanup" Nov 03 17:45:48.545833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.545798 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.617826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.617785 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.617826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.617827 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618021 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.617850 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618021 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.617955 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618021 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.617993 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8rml\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618021 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.618017 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618180 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.618098 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618180 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.618147 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.618180 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.618167 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.667416 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.667383 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:45:48.719642 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719565 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719642 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719645 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8rml\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719665 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719700 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719748 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719774 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719816 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719839 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.719880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.719858 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.720235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.720043 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.720235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.720096 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.720597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.720540 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.720715 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.720545 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.720772 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.720733 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.722374 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.722346 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.724371 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.724348 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.784054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.784020 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.784218 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.784130 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8rml\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\") pod \"maas-default-gateway-openshift-default-5df48fcc99-qs9s4\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:48.860042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:48.859957 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:49.295078 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:49.295021 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:45:49.298981 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:45:49.298946 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod136e96f2_2bd9_4fed_9bc7_152b9bb9827d.slice/crio-87bcd2ff0b43afad0229a3cbc63a68f7978963d4c776556a752ea87804b5f72d WatchSource:0}: Error finding container 87bcd2ff0b43afad0229a3cbc63a68f7978963d4c776556a752ea87804b5f72d: Status 404 returned error can't find the container with id 87bcd2ff0b43afad0229a3cbc63a68f7978963d4c776556a752ea87804b5f72d Nov 03 17:45:49.306268 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:49.306236 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:49.306350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:49.306306 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:49.306350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:49.306335 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:45:49.318141 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:49.318115 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" event={"ID":"136e96f2-2bd9-4fed-9bc7-152b9bb9827d","Type":"ContainerStarted","Data":"87bcd2ff0b43afad0229a3cbc63a68f7978963d4c776556a752ea87804b5f72d"} Nov 03 17:45:50.324872 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:50.324838 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" event={"ID":"136e96f2-2bd9-4fed-9bc7-152b9bb9827d","Type":"ContainerStarted","Data":"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb"} Nov 03 17:45:50.509319 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:50.509252 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podStartSLOduration=2.509232148 podStartE2EDuration="2.509232148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:45:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:45:50.508824765 +0000 UTC m=+730.337908368" watchObservedRunningTime="2026-11-03 17:45:50.509232148 +0000 UTC m=+730.338315752" Nov 03 17:45:50.871598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:50.862920 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:50.879760 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:50.879726 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:50.879933 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:50.879787 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:51.861968 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:51.861932 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:51.862612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:51.862557 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:52.873600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:52.865968 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:52.873600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:52.866041 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:53.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:53.877976 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:53.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:53.878043 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:54.868053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:54.868015 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:54.868346 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:54.868315 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:55.861113 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:55.861080 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:55.861497 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:55.861133 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:56.860785 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:56.860748 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-qs9s4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" start-of-body= Nov 03 17:45:56.860945 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:56.860825 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.46:15021/healthz/ready\": dial tcp 10.132.0.46:15021: connect: connection refused" Nov 03 17:45:58.072272 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:58.072226 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:58.072804 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:58.072783 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:45:58.073682 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:45:58.073660 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:46:14.519046 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:46:14.519010 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:46:44.518548 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:46:44.518515 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:46:48.309130 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.309091 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 17:46:48.312903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.312881 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.387688 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.387652 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 17:46:48.396703 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396676 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.396833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396722 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.396833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396749 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.396907 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396892 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.396963 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396945 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.396998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.396980 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.397030 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.397005 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhdst\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.397094 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.397074 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.397226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.397205 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.416385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.416364 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7d463-openshift-default-dockercfg-wqnxq\"" Nov 03 17:46:48.497660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497629 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.497818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497683 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.497818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497706 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.497818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497727 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhdst\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.497818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497744 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.497818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497773 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497819 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497858 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.497886 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.498073 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498265 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.498202 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498454 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.498430 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.498687 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.498812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.498738 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.500213 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.500192 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.500551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.500530 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.560445 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.560378 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.560598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.560560 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhdst\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.625691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.625654 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:48.983860 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.983821 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 17:46:48.988462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:48.988441 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.043254 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.043224 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-c76280-openshift-default-dockercfg-mtxr6\"" Nov 03 17:46:49.051189 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.051162 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 17:46:49.104084 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104049 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104094 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104119 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104269 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104317 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104459 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104380 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104459 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104408 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104536 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104508 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5csq\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.104600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.104548 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.147166 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.147142 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 17:46:49.149701 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:46:49.149668 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1074adce_5d3a_4974_9ec1_c40f5abc4bbe.slice/crio-d99be589bd99610b88458257fe309c62aa5ba42009f88ae8def774612fe5d4b5 WatchSource:0}: Error finding container d99be589bd99610b88458257fe309c62aa5ba42009f88ae8def774612fe5d4b5: Status 404 returned error can't find the container with id d99be589bd99610b88458257fe309c62aa5ba42009f88ae8def774612fe5d4b5 Nov 03 17:46:49.151905 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.151875 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.152002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.151942 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.152002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.151969 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.206036 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206009 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206054 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206077 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206102 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206142 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206349 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206231 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206349 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206269 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206349 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206326 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5csq\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206350 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206452 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206654 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206794 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206767 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206849 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206772 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.206849 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.206819 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.208719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.208699 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.209090 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.209069 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.269593 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.269500 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.269724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.269607 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5csq\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.313900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.313863 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:49.602517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.602440 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerStarted","Data":"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd"} Nov 03 17:46:49.602517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.602489 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerStarted","Data":"d99be589bd99610b88458257fe309c62aa5ba42009f88ae8def774612fe5d4b5"} Nov 03 17:46:49.626107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.626082 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:49.654236 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.654207 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 17:46:49.659076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.659048 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713271 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713225 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713280 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713314 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713339 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713366 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713405 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hr8v\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713462 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713843 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713501 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.713843 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.713565 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.718150 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.718125 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-73a606-openshift-default-dockercfg-6xzts\"" Nov 03 17:46:49.733223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.730316 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 17:46:49.733223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.733175 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podStartSLOduration=1.733159128 podStartE2EDuration="1.733159128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:49.729781665 +0000 UTC m=+789.558865472" watchObservedRunningTime="2026-11-03 17:46:49.733159128 +0000 UTC m=+789.562242731" Nov 03 17:46:49.764187 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.764156 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 17:46:49.766239 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:46:49.766215 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4117524c_c6d8_46d1_8516_9138033f2b06.slice/crio-b1b1f46f3a79c8fa7029dbeb43f72c3f598f7344514347a1c3918fcec25ae81a WatchSource:0}: Error finding container b1b1f46f3a79c8fa7029dbeb43f72c3f598f7344514347a1c3918fcec25ae81a: Status 404 returned error can't find the container with id b1b1f46f3a79c8fa7029dbeb43f72c3f598f7344514347a1c3918fcec25ae81a Nov 03 17:46:49.768914 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.768885 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.769007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.768944 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.769007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.768972 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:49.816289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816259 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816303 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816340 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816365 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816473 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816525 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hr8v\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816600 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816701 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816747 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816769 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.816941 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.816829 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.817073 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.817055 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.817264 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.817240 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.817345 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.817272 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.819103 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.819080 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.819672 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.819653 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.885443 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.885318 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hr8v\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.890153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.890086 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:49.975443 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:49.975410 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:50.415560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.415526 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 17:46:50.418318 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:46:50.418287 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ecb1af_07b7_474a_a3c2_a2eb50ddc736.slice/crio-6c88c7473d9e75df04a314c18970ee6963e313058964978ba7e25faf26f9e9ee WatchSource:0}: Error finding container 6c88c7473d9e75df04a314c18970ee6963e313058964978ba7e25faf26f9e9ee: Status 404 returned error can't find the container with id 6c88c7473d9e75df04a314c18970ee6963e313058964978ba7e25faf26f9e9ee Nov 03 17:46:50.421078 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.421045 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:50.421158 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.421116 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:50.421158 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.421150 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:46:50.608521 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.608478 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerStarted","Data":"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047"} Nov 03 17:46:50.608753 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.608542 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerStarted","Data":"b1b1f46f3a79c8fa7029dbeb43f72c3f598f7344514347a1c3918fcec25ae81a"} Nov 03 17:46:50.610410 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.610377 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerStarted","Data":"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0"} Nov 03 17:46:50.610532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.610435 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerStarted","Data":"6c88c7473d9e75df04a314c18970ee6963e313058964978ba7e25faf26f9e9ee"} Nov 03 17:46:50.631734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.631710 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:50.738087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.738037 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podStartSLOduration=2.738021852 podStartE2EDuration="2.738021852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:50.73240533 +0000 UTC m=+790.561488943" watchObservedRunningTime="2026-11-03 17:46:50.738021852 +0000 UTC m=+790.567105455" Nov 03 17:46:50.857975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.857921 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podStartSLOduration=1.857906593 podStartE2EDuration="1.857906593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:46:50.85565482 +0000 UTC m=+790.684738422" watchObservedRunningTime="2026-11-03 17:46:50.857906593 +0000 UTC m=+790.686990194" Nov 03 17:46:50.975982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:50.975948 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:51.314878 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.314846 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:51.320233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.320208 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:51.615941 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.615847 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:51.616394 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.616110 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:51.616871 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.616847 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:46:51.617237 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.617220 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:46:51.982762 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:51.982732 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:52.620969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:52.620936 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:52.621639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:52.621618 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:46:58.628285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.628253 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 17:46:58.633100 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.633077 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.688997 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.688975 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-07f297\"" Nov 03 17:46:58.719459 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.719424 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.719620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.719478 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzpjn\" (UniqueName: \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.741056 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.741005 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-07f297-dockercfg-rz2fm\"" Nov 03 17:46:58.744223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.744138 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 17:46:58.820343 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.820303 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.820534 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.820355 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzpjn\" (UniqueName: \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.821123 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.821103 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.883412 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.883344 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzpjn\" (UniqueName: \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\") pod \"payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:58.947163 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:58.947120 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:46:59.260119 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.260079 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 17:46:59.264968 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.264950 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.319742 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.319708 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-f7d463-serving-cert\"" Nov 03 17:46:59.319891 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.319787 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 17:46:59.319891 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.319807 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fhmsl\"" Nov 03 17:46:59.325948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.325917 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.326088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.326069 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.326152 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.326113 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmk45\" (UniqueName: \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.326152 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.326122 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 17:46:59.326227 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.326196 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.375142 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.375118 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 17:46:59.427708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.427675 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.427884 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.427716 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmk45\" (UniqueName: \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.427884 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.427743 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.427884 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.427796 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.427884 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:46:59.427839 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-serving-cert: secret "maas-api-e2e-worker-w6-f7d463-serving-cert" not found Nov 03 17:46:59.428056 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:46:59.427911 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls podName:022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c nodeName:}" failed. No retries permitted until 2026-11-03 17:46:59.927889309 +0000 UTC m=+799.756972908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls") pod "maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" (UID: "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c") : secret "maas-api-e2e-worker-w6-f7d463-serving-cert" not found Nov 03 17:46:59.428426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.428406 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.430340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.430323 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.491320 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.491296 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmk45\" (UniqueName: \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.655149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.655060 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" event={"ID":"e7bc4f39-5ff4-411d-85cf-a248ec146123","Type":"ContainerStarted","Data":"4e973696922cc0100c1f603b280668ee255a37fed2133e55cd69c3082982c812"} Nov 03 17:46:59.933774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.933698 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:46:59.936341 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:46:59.936319 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") pod \"maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:47:00.177975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:00.177943 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:47:00.558692 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:00.558664 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 17:47:00.560885 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:00.560855 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod022cffd6_d3ab_4eee_ba6a_f7d8be2f3c4c.slice/crio-c21f923bcf50de147b48127e7873ce2fd2f20b81f072ad8ca55e61a069707899 WatchSource:0}: Error finding container c21f923bcf50de147b48127e7873ce2fd2f20b81f072ad8ca55e61a069707899: Status 404 returned error can't find the container with id c21f923bcf50de147b48127e7873ce2fd2f20b81f072ad8ca55e61a069707899 Nov 03 17:47:00.661741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:00.661706 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" event={"ID":"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c","Type":"ContainerStarted","Data":"c21f923bcf50de147b48127e7873ce2fd2f20b81f072ad8ca55e61a069707899"} Nov 03 17:47:01.677551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:01.677498 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" event={"ID":"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c","Type":"ContainerStarted","Data":"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7"} Nov 03 17:47:01.677975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:01.677616 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:47:01.799826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:01.799762 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" podStartSLOduration=1.928917671 podStartE2EDuration="2.799740896s" totalImagesPullingTime="870.823225ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:01.798375777 +0000 UTC m=+801.627459381" watchObservedRunningTime="2026-11-03 17:47:01.799740896 +0000 UTC m=+801.628824496" Nov 03 17:47:02.684388 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:02.684348 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" event={"ID":"e7bc4f39-5ff4-411d-85cf-a248ec146123","Type":"ContainerStarted","Data":"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964"} Nov 03 17:47:02.684839 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:02.684616 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:47:02.805325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:02.805270 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" podStartSLOduration=2.208765311 podStartE2EDuration="4.80525596s" totalImagesPullingTime="2.596490649s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:02.803945082 +0000 UTC m=+802.633028684" watchObservedRunningTime="2026-11-03 17:47:02.80525596 +0000 UTC m=+802.634339556" Nov 03 17:47:04.776850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.776810 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 17:47:04.781129 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.781105 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.834396 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.834363 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w3-776ba7-serving-cert\"" Nov 03 17:47:04.883269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.883227 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.883438 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.883274 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.883438 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.883390 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fq5\" (UniqueName: \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.883565 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.883438 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.886510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.886488 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 17:47:04.984768 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.984728 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.984768 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.984771 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.985011 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:04.984882 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-serving-cert: secret "maas-api-e2e-worker-w3-776ba7-serving-cert" not found Nov 03 17:47:04.985011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.984909 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fq5\" (UniqueName: \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.985011 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:04.984938 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls podName:b46938a7-f0fe-4d64-9f98-04b73326b51c nodeName:}" failed. No retries permitted until 2026-11-03 17:47:05.484921514 +0000 UTC m=+805.314005098 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls") pod "maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" (UID: "b46938a7-f0fe-4d64-9f98-04b73326b51c") : secret "maas-api-e2e-worker-w3-776ba7-serving-cert" not found Nov 03 17:47:04.985011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.984985 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.985502 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.985479 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:04.987478 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:04.987456 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:05.048169 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:05.048111 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fq5\" (UniqueName: \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:05.490838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:05.490793 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:05.493329 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:05.493309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") pod \"maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:05.695253 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:05.695215 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:06.078948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:06.078918 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 17:47:06.080283 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:06.080257 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb46938a7_f0fe_4d64_9f98_04b73326b51c.slice/crio-37cfd06fc250d98fa41a1cf84e23768f348d729802cdb00b2475a76b3ea0f94a WatchSource:0}: Error finding container 37cfd06fc250d98fa41a1cf84e23768f348d729802cdb00b2475a76b3ea0f94a: Status 404 returned error can't find the container with id 37cfd06fc250d98fa41a1cf84e23768f348d729802cdb00b2475a76b3ea0f94a Nov 03 17:47:06.705741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:06.705706 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" event={"ID":"b46938a7-f0fe-4d64-9f98-04b73326b51c","Type":"ContainerStarted","Data":"37cfd06fc250d98fa41a1cf84e23768f348d729802cdb00b2475a76b3ea0f94a"} Nov 03 17:47:07.691076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:07.691043 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 17:47:07.713601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:07.713545 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" event={"ID":"b46938a7-f0fe-4d64-9f98-04b73326b51c","Type":"ContainerStarted","Data":"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd"} Nov 03 17:47:07.713765 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:07.713637 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:07.939882 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:07.939811 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" podStartSLOduration=3.136647056 podStartE2EDuration="3.939787833s" totalImagesPullingTime="803.140777ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:07.934407174 +0000 UTC m=+807.763490780" watchObservedRunningTime="2026-11-03 17:47:07.939787833 +0000 UTC m=+807.768871438" Nov 03 17:47:13.693128 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.693091 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 17:47:13.741629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.741596 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 17:47:13.868341 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.868306 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 17:47:13.873071 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.873045 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:13.932466 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.932437 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-c76280\"" Nov 03 17:47:13.932679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.932505 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-c76280-dockercfg-ll68d\"" Nov 03 17:47:13.934318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.934291 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 17:47:13.941828 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.941798 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:13.941977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:13.941841 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsmdq\" (UniqueName: \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.042970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.042885 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.042970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.042944 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsmdq\" (UniqueName: \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.043612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.043594 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.107921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.107884 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsmdq\" (UniqueName: \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\") pod \"payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.187598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.187536 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.519887 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:14.519844 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:47:14.567828 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.567791 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 17:47:14.569929 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:14.569901 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648af06e_e212_42f7_a85b_62a350a6776f.slice/crio-4f5e5a9516672dd89cb7ee57a9f0aead32c2129ef3209b5b41e84ab77d13d0ac WatchSource:0}: Error finding container 4f5e5a9516672dd89cb7ee57a9f0aead32c2129ef3209b5b41e84ab77d13d0ac: Status 404 returned error can't find the container with id 4f5e5a9516672dd89cb7ee57a9f0aead32c2129ef3209b5b41e84ab77d13d0ac Nov 03 17:47:14.761243 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.761204 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" event={"ID":"648af06e-e212-42f7-a85b-62a350a6776f","Type":"ContainerStarted","Data":"32cf9118c17ebf7e49fed046aaee339142bae0c7b7d030231c382dc618dbbfc6"} Nov 03 17:47:14.761698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.761271 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" event={"ID":"648af06e-e212-42f7-a85b-62a350a6776f","Type":"ContainerStarted","Data":"4f5e5a9516672dd89cb7ee57a9f0aead32c2129ef3209b5b41e84ab77d13d0ac"} Nov 03 17:47:14.761698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.761387 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:14.882491 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:14.882368 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" podStartSLOduration=1.882348743 podStartE2EDuration="1.882348743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:14.880232553 +0000 UTC m=+814.709316157" watchObservedRunningTime="2026-11-03 17:47:14.882348743 +0000 UTC m=+814.711432346" Nov 03 17:47:16.432368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.432332 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 17:47:16.436479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.436454 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.467278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.467251 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.467462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.467436 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf684\" (UniqueName: \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.491049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.491021 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-548ca1\"" Nov 03 17:47:16.540287 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.540262 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-548ca1-dockercfg-7gknk\"" Nov 03 17:47:16.543623 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.543600 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 17:47:16.568054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.568027 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf684\" (UniqueName: \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.568201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.568073 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.568877 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.568858 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.633010 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.632957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf684\" (UniqueName: \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\") pod \"payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:16.749930 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:16.749895 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:17.143283 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.143154 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 17:47:17.146422 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:17.146388 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b09eb49_c2df_4df4_99cf_c5d1594dec9a.slice/crio-67f8559490cbd2c54861aa4377069e0ce0e8c584e1788f49b139b7ec52815aff WatchSource:0}: Error finding container 67f8559490cbd2c54861aa4377069e0ce0e8c584e1788f49b139b7ec52815aff: Status 404 returned error can't find the container with id 67f8559490cbd2c54861aa4377069e0ce0e8c584e1788f49b139b7ec52815aff Nov 03 17:47:17.783310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.783210 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" event={"ID":"3b09eb49-c2df-4df4-99cf-c5d1594dec9a","Type":"ContainerStarted","Data":"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a"} Nov 03 17:47:17.783310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.783267 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" event={"ID":"3b09eb49-c2df-4df4-99cf-c5d1594dec9a","Type":"ContainerStarted","Data":"67f8559490cbd2c54861aa4377069e0ce0e8c584e1788f49b139b7ec52815aff"} Nov 03 17:47:17.783310 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.783304 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:17.784765 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.784741 4980 generic.go:356] "Generic (PLEG): container finished" podID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerID="ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd" exitCode=0 Nov 03 17:47:17.784873 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.784808 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerDied","Data":"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd"} Nov 03 17:47:17.785115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.785101 4980 scope.go:122] "RemoveContainer" containerID="ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd" Nov 03 17:47:17.786362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.786336 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:17.786441 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.786403 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:17.786480 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.786451 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:17.904831 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:17.904774 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" podStartSLOduration=1.904759925 podStartE2EDuration="1.904759925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:17.902521002 +0000 UTC m=+817.731604605" watchObservedRunningTime="2026-11-03 17:47:17.904759925 +0000 UTC m=+817.733843526" Nov 03 17:47:18.791368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:18.791331 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerStarted","Data":"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262"} Nov 03 17:47:19.626781 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:19.626743 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:47:19.627797 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:19.627773 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 17:47:19.627918 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:19.627817 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 17:47:20.625984 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.625951 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 17:47:20.626462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.626002 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 17:47:20.866600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.866320 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 17:47:20.886694 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.886625 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:20.918340 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.918310 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:20.918472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.918346 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fknd2\" (UniqueName: \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:20.918472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.918465 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:20.918612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.918495 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:20.934518 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.934491 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 17:47:20.983663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:20.982316 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w1-c76280-serving-cert\"" Nov 03 17:47:21.019413 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.019376 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.019627 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.019430 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.019627 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.019507 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.019627 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.019530 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fknd2\" (UniqueName: \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.019805 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:21.019650 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-serving-cert: secret "maas-api-e2e-worker-w1-c76280-serving-cert" not found Nov 03 17:47:21.019805 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:21.019724 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls podName:ab978c3e-1202-4ffb-8281-cfb352c1741b nodeName:}" failed. No retries permitted until 2026-11-03 17:47:21.519702599 +0000 UTC m=+821.348786194 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls") pod "maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" (UID: "ab978c3e-1202-4ffb-8281-cfb352c1741b") : secret "maas-api-e2e-worker-w1-c76280-serving-cert" not found Nov 03 17:47:21.020030 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.020007 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.022384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.022357 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.086897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.086865 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fknd2\" (UniqueName: \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.550742 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.550699 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.553251 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.553217 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") pod \"maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:21.615861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.615823 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:47:21.626020 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.625981 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 17:47:21.626406 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.626040 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 17:47:21.799780 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:21.799746 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:22.251224 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:22.251195 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 17:47:22.253277 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:22.253247 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab978c3e_1202_4ffb_8281_cfb352c1741b.slice/crio-7a98b43f396a6594f0e3208b2e14fd127e9a03d092bd6fbfc4b6fcf8e12ffb9b WatchSource:0}: Error finding container 7a98b43f396a6594f0e3208b2e14fd127e9a03d092bd6fbfc4b6fcf8e12ffb9b: Status 404 returned error can't find the container with id 7a98b43f396a6594f0e3208b2e14fd127e9a03d092bd6fbfc4b6fcf8e12ffb9b Nov 03 17:47:22.627031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:22.626951 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 17:47:22.627031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:22.627001 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 17:47:22.812644 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:22.812608 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" event={"ID":"ab978c3e-1202-4ffb-8281-cfb352c1741b","Type":"ContainerStarted","Data":"7a98b43f396a6594f0e3208b2e14fd127e9a03d092bd6fbfc4b6fcf8e12ffb9b"} Nov 03 17:47:23.626866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:23.626834 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" start-of-body= Nov 03 17:47:23.627061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:23.626893 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": dial tcp 10.132.0.47:15021: connect: connection refused" Nov 03 17:47:23.819302 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:23.819264 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" event={"ID":"ab978c3e-1202-4ffb-8281-cfb352c1741b","Type":"ContainerStarted","Data":"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f"} Nov 03 17:47:23.819479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:23.819355 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:23.944537 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:23.944478 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" podStartSLOduration=3.069111032 podStartE2EDuration="3.944459571s" totalImagesPullingTime="875.348539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:23.944033652 +0000 UTC m=+823.773117254" watchObservedRunningTime="2026-11-03 17:47:23.944459571 +0000 UTC m=+823.773543174" Nov 03 17:47:24.630484 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:24.630452 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:47:24.631474 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:24.631454 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 17:47:25.770036 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:25.770004 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 17:47:28.795278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:28.795246 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 17:47:29.829631 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:29.829594 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 17:47:37.527483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.526174 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:46080->10.132.0.48:15021: read: connection reset by peer" start-of-body= Nov 03 17:47:37.527483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.526294 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": read tcp 10.132.0.2:46080->10.132.0.48:15021: read: connection reset by peer" Nov 03 17:47:37.916946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.916867 4980 generic.go:356] "Generic (PLEG): container finished" podID="4117524c-c6d8-46d1-8516-9138033f2b06" containerID="6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047" exitCode=0 Nov 03 17:47:37.916946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.916909 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerDied","Data":"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047"} Nov 03 17:47:37.917291 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.917276 4980 scope.go:122] "RemoveContainer" containerID="6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047" Nov 03 17:47:37.918532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.918508 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:37.918628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.918594 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:37.918676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:37.918628 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:47:38.923757 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:38.923722 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerStarted","Data":"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76"} Nov 03 17:47:39.315011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:39.314976 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:47:39.315331 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:39.315308 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:47:39.315389 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:39.315363 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 17:47:40.314873 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:40.314842 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:47:40.315250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:40.314905 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 17:47:41.314474 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:41.314442 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:47:41.314674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:41.314490 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 17:47:42.183630 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.183594 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 17:47:42.188417 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.188398 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.222651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.222625 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.222763 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.222661 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l8lh\" (UniqueName: \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.222841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.222825 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.222913 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.222889 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.232433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.232409 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 17:47:42.282819 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.282794 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-73a606-serving-cert\"" Nov 03 17:47:42.314660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.314628 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:47:42.314826 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.314685 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 17:47:42.324128 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.324105 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l8lh\" (UniqueName: \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.324242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.324227 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.324309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.324260 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.324364 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.324343 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.324416 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:42.324368 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-serving-cert: secret "maas-api-e2e-worker-w4-73a606-serving-cert" not found Nov 03 17:47:42.324467 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:42.324444 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls podName:4a5f7daa-bc77-49d6-af16-52075beb0338 nodeName:}" failed. No retries permitted until 2026-11-03 17:47:42.824412109 +0000 UTC m=+842.653495689 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls") pod "maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" (UID: "4a5f7daa-bc77-49d6-af16-52075beb0338") : secret "maas-api-e2e-worker-w4-73a606-serving-cert" not found Nov 03 17:47:42.324982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.324957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.326887 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.326869 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.387463 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.387432 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l8lh\" (UniqueName: \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.829034 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.828988 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:42.831689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:42.831661 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") pod \"maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:43.102223 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:43.102150 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:43.318799 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:43.318765 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body= Nov 03 17:47:43.319163 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:43.318834 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" Nov 03 17:47:43.536211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:43.536181 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 17:47:43.537565 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:47:43.537536 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5f7daa_bc77_49d6_af16_52075beb0338.slice/crio-485275b328a7ccb00ef75d5d68811c539e44f841a337997153f7b9c7c8294446 WatchSource:0}: Error finding container 485275b328a7ccb00ef75d5d68811c539e44f841a337997153f7b9c7c8294446: Status 404 returned error can't find the container with id 485275b328a7ccb00ef75d5d68811c539e44f841a337997153f7b9c7c8294446 Nov 03 17:47:43.949333 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:43.949301 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" event={"ID":"4a5f7daa-bc77-49d6-af16-52075beb0338","Type":"ContainerStarted","Data":"485275b328a7ccb00ef75d5d68811c539e44f841a337997153f7b9c7c8294446"} Nov 03 17:47:44.318476 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:44.318363 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:47:44.318676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:44.318653 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:47:44.319404 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:44.319383 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 17:47:44.520730 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:47:44.520696 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:47:44.956841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:44.956783 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" event={"ID":"4a5f7daa-bc77-49d6-af16-52075beb0338","Type":"ContainerStarted","Data":"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1"} Nov 03 17:47:44.957017 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:44.956883 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:47:45.080063 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:45.079954 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" podStartSLOduration=2.25539069 podStartE2EDuration="3.07993643s" totalImagesPullingTime="824.54574ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:47:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:47:45.079849035 +0000 UTC m=+844.908932649" watchObservedRunningTime="2026-11-03 17:47:45.07993643 +0000 UTC m=+844.909020034" Nov 03 17:47:50.968913 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:47:50.968836 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 17:48:03.056276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.056239 4980 generic.go:356] "Generic (PLEG): container finished" podID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerID="d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0" exitCode=0 Nov 03 17:48:03.056684 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.056288 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerDied","Data":"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0"} Nov 03 17:48:03.056736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.056720 4980 scope.go:122] "RemoveContainer" containerID="d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0" Nov 03 17:48:03.058193 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.058167 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:03.058259 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.058236 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:03.058297 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:03.058271 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:48:04.087598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:04.081015 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerStarted","Data":"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b"} Nov 03 17:48:04.977607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:04.977219 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:48:04.977607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:04.977352 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:48:04.977607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:04.977408 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 17:48:05.976952 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:05.976915 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:48:05.979379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:05.979345 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 17:48:06.979599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:06.975952 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:48:06.979599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:06.976017 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 17:48:07.620961 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:07.620924 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:48:07.978598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:07.978027 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:48:07.978598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:07.978092 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 17:48:08.979600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:08.977001 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" start-of-body= Nov 03 17:48:08.979600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:08.977080 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.49:15021/healthz/ready\": dial tcp 10.132.0.49:15021: connect: connection refused" Nov 03 17:48:09.980801 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:09.980770 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:48:09.981899 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:09.981878 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 17:48:14.520023 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:48:14.519986 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:48:40.373380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:40.373351 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:48:40.375359 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:40.375336 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:48:40.379016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:40.378991 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:48:40.381002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:48:40.380984 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:48:44.519241 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:48:44.519198 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:49:14.518296 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:49:14.518257 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:49:18.195790 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.195744 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 17:49:18.199891 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.199866 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.254151 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.254118 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"openshift-service-ca.crt\"" Nov 03 17:49:18.296806 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.296774 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.296966 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.296880 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.296966 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.296908 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf87m\" (UniqueName: \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.296966 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.296958 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.297076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.296982 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.297112 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.297076 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.303105 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.303080 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"kube-root-ca.crt\"" Nov 03 17:49:18.303322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.303308 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"default-dockercfg-dv5pf\"" Nov 03 17:49:18.303420 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.303365 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-776ba7\"/\"premium-w3-776ba7-kserve-self-signed-certs\"" Nov 03 17:49:18.305610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.305580 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 17:49:18.398447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398404 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.398674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398488 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.398674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398520 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf87m\" (UniqueName: \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.398674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398538 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.398857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398741 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.398963 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.398940 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.399048 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.399030 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.399249 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.399168 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.399800 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.399773 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.401323 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.401299 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.401487 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.401453 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.465033 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.464959 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf87m\" (UniqueName: \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\") pod \"premium-w3-776ba7-kserve-5c68559945-jdv9c\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.512114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.512075 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:18.910204 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:18.910176 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 17:49:18.912155 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:49:18.912127 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafee383f_b486_4de5_a58f_a30e14a768ee.slice/crio-6c27c24c7ce9fc1ccfa4441c337dc4a1e942b636d32b2da32a3ba062de53a3d5 WatchSource:0}: Error finding container 6c27c24c7ce9fc1ccfa4441c337dc4a1e942b636d32b2da32a3ba062de53a3d5: Status 404 returned error can't find the container with id 6c27c24c7ce9fc1ccfa4441c337dc4a1e942b636d32b2da32a3ba062de53a3d5 Nov 03 17:49:19.539564 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:19.539525 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" event={"ID":"afee383f-b486-4de5-a58f-a30e14a768ee","Type":"ContainerStarted","Data":"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63"} Nov 03 17:49:19.539986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:19.539596 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" event={"ID":"afee383f-b486-4de5-a58f-a30e14a768ee","Type":"ContainerStarted","Data":"6c27c24c7ce9fc1ccfa4441c337dc4a1e942b636d32b2da32a3ba062de53a3d5"} Nov 03 17:49:19.663133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:19.663071 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podStartSLOduration=1.663053177 podStartE2EDuration="1.663053177s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:19.660553848 +0000 UTC m=+939.489637450" watchObservedRunningTime="2026-11-03 17:49:19.663053177 +0000 UTC m=+939.492136779" Nov 03 17:49:20.513076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:20.513003 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:23.099435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.099396 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 17:49:23.111007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.110985 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.152771 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.152739 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.152935 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.152856 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.152935 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.152912 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.153019 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.152942 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.153074 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.153050 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.153150 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.153133 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq257\" (UniqueName: \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.163476 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.163447 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 17:49:23.210890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.210863 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"simulator-w5-548ca1-kserve-self-signed-certs\"" Nov 03 17:49:23.211037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.210957 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"default-dockercfg-cqnvp\"" Nov 03 17:49:23.211037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.211015 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"openshift-service-ca.crt\"" Nov 03 17:49:23.211037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.211017 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-548ca1\"/\"kube-root-ca.crt\"" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254273 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254342 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254396 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254475 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq257\" (UniqueName: \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254508 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.254663 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.255326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.255154 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.256510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.256481 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.256850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.256822 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.257920 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.257895 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.258817 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.258789 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.319429 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.319402 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq257\" (UniqueName: \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\") pod \"simulator-w5-548ca1-kserve-5894b98ddb-fxlsz\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.426886 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.426808 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:23.808538 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:23.808512 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 17:49:23.809704 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:49:23.809674 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf38ea01_f628_4f52_91e4_58d8f2103bc4.slice/crio-dd6ccfa6c45fd7b4932eeeced9796ff32d6e7a595546f874702817d24cd43ce9 WatchSource:0}: Error finding container dd6ccfa6c45fd7b4932eeeced9796ff32d6e7a595546f874702817d24cd43ce9: Status 404 returned error can't find the container with id dd6ccfa6c45fd7b4932eeeced9796ff32d6e7a595546f874702817d24cd43ce9 Nov 03 17:49:24.571747 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:24.571713 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" event={"ID":"cf38ea01-f628-4f52-91e4-58d8f2103bc4","Type":"ContainerStarted","Data":"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8"} Nov 03 17:49:24.572149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:24.571757 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" event={"ID":"cf38ea01-f628-4f52-91e4-58d8f2103bc4","Type":"ContainerStarted","Data":"dd6ccfa6c45fd7b4932eeeced9796ff32d6e7a595546f874702817d24cd43ce9"} Nov 03 17:49:24.692977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:24.692922 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podStartSLOduration=1.6929077860000001 podStartE2EDuration="1.692907786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:49:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:49:24.691913907 +0000 UTC m=+944.520997509" watchObservedRunningTime="2026-11-03 17:49:24.692907786 +0000 UTC m=+944.521991417" Nov 03 17:49:25.427180 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:25.427152 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:28.512954 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:28.512921 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:28.525333 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:28.525310 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:28.603383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:28.603360 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:49:33.427452 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:33.427415 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:33.447034 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:33.446727 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:33.630782 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:49:33.630755 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:49:44.519407 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:49:44.519371 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:50:02.896918 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.896876 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 17:50:02.900908 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.900889 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.955536 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.955510 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"openshift-service-ca.crt\"" Nov 03 17:50:02.984129 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984097 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.984280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984140 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.984280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984158 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.984280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984256 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6m8n\" (UniqueName: \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.984392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984307 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:02.984392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:02.984341 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.004634 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.004603 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"kube-root-ca.crt\"" Nov 03 17:50:03.004764 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.004719 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"default-dockercfg-hr25s\"" Nov 03 17:50:03.004764 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.004744 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"premium-w1-c76280-kserve-self-signed-certs\"" Nov 03 17:50:03.007720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.007700 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 17:50:03.085922 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.085891 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.085943 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.085975 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086039 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6m8n\" (UniqueName: \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086288 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086119 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086288 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086144 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086390 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086333 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086390 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086351 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.086552 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.086529 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.088370 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.088345 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.088829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.088809 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.149639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.149559 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6m8n\" (UniqueName: \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\") pod \"premium-w1-c76280-kserve-64ccf4ff5f-kbsk8\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.213453 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.213413 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:03.623319 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.623287 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 17:50:03.625250 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:50:03.625221 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a49f302_04b3_4ff0_b880_fde16f2c4b4a.slice/crio-76bf32aa4d5e6229c45b01ea4bce09293b46b710f26919fd7e36bd0e2e1941e0 WatchSource:0}: Error finding container 76bf32aa4d5e6229c45b01ea4bce09293b46b710f26919fd7e36bd0e2e1941e0: Status 404 returned error can't find the container with id 76bf32aa4d5e6229c45b01ea4bce09293b46b710f26919fd7e36bd0e2e1941e0 Nov 03 17:50:03.783324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.783285 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" event={"ID":"2a49f302-04b3-4ff0-b880-fde16f2c4b4a","Type":"ContainerStarted","Data":"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c"} Nov 03 17:50:03.783324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.783326 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" event={"ID":"2a49f302-04b3-4ff0-b880-fde16f2c4b4a","Type":"ContainerStarted","Data":"76bf32aa4d5e6229c45b01ea4bce09293b46b710f26919fd7e36bd0e2e1941e0"} Nov 03 17:50:03.904195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:03.904099 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podStartSLOduration=1.9040852350000002 podStartE2EDuration="1.904085235s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:50:03.90247305 +0000 UTC m=+983.731556653" watchObservedRunningTime="2026-11-03 17:50:03.904085235 +0000 UTC m=+983.733168840" Nov 03 17:50:04.213801 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:04.213764 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:13.213651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:13.213615 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:13.227137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:13.227111 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:13.848716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:13.848685 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:50:14.519898 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:50:14.519860 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:50:44.519848 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:50:44.519806 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:50:50.914875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:50.914655 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:50:50.915591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:50.915525 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" containerID="cri-o://6f8a54bd293fb7cc9617b34ab93ca0bdfbfcb52fe6ae24aabae140df6ee9bda9" gracePeriod=30 Nov 03 17:50:56.078257 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.078221 4980 generic.go:356] "Generic (PLEG): container finished" podID="134977fe-81ee-45dc-8d23-b76703e17683" containerID="6f8a54bd293fb7cc9617b34ab93ca0bdfbfcb52fe6ae24aabae140df6ee9bda9" exitCode=0 Nov 03 17:50:56.078732 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.078295 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" event={"ID":"134977fe-81ee-45dc-8d23-b76703e17683","Type":"ContainerDied","Data":"6f8a54bd293fb7cc9617b34ab93ca0bdfbfcb52fe6ae24aabae140df6ee9bda9"} Nov 03 17:50:56.177979 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.177907 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:50:56.284993 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.284962 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.284998 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285022 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285081 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285118 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285210 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285273 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:50:56.285379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285316 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285373 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285369 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:50:56.285612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285428 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\") pod \"134977fe-81ee-45dc-8d23-b76703e17683\" (UID: \"134977fe-81ee-45dc-8d23-b76703e17683\") " Nov 03 17:50:56.285612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285603 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:50:56.285773 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285627 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:50:56.285874 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285850 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:50:56.285958 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285927 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.285958 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285950 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/134977fe-81ee-45dc-8d23-b76703e17683-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.286052 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285964 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.286052 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.285978 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.287694 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.287666 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:56.287908 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.287884 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "kube-api-access-mzkdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:50:56.288242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.288216 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:50:56.288368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.288335 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo" pod "134977fe-81ee-45dc-8d23-b76703e17683" (UID: "134977fe-81ee-45dc-8d23-b76703e17683"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:50:56.386911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.386881 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/134977fe-81ee-45dc-8d23-b76703e17683-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.386911 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.386911 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.387107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.386925 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzkdm\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-kube-api-access-mzkdm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.387107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.386940 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/134977fe-81ee-45dc-8d23-b76703e17683-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:56.387107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:56.386952 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/134977fe-81ee-45dc-8d23-b76703e17683-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:50:57.084664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:57.084624 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" event={"ID":"134977fe-81ee-45dc-8d23-b76703e17683","Type":"ContainerDied","Data":"4d57330a85ca2816a1466948b1af04d94d1373d782ddd0e16f1f5a39345abe80"} Nov 03 17:50:57.084664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:57.084653 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k" Nov 03 17:50:57.085211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:57.084688 4980 scope.go:122] "RemoveContainer" containerID="6f8a54bd293fb7cc9617b34ab93ca0bdfbfcb52fe6ae24aabae140df6ee9bda9" Nov 03 17:50:57.304421 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:57.304393 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:50:57.354121 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:57.354049 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-zgg6k"] Nov 03 17:50:58.292996 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:50:58.292955 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="134977fe-81ee-45dc-8d23-b76703e17683" path="/var/lib/kubelet/pods/134977fe-81ee-45dc-8d23-b76703e17683/volumes" Nov 03 17:51:14.520227 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:51:14.520187 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:51:36.100355 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.100315 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 17:51:36.100955 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.100832 4980 cpu_manager.go:414] "containerMap: removing container" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" Nov 03 17:51:36.100955 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.100844 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" Nov 03 17:51:36.100955 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.100955 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="134977fe-81ee-45dc-8d23-b76703e17683" containerName="istio-proxy" Nov 03 17:51:36.104449 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.104426 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.159254 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.159210 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-776ba7-openshift-default-dockercfg-nv8qc\"" Nov 03 17:51:36.211995 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.211959 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 17:51:36.212424 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212402 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212486 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212443 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjpdv\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212486 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212471 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212681 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212658 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212740 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212724 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212835 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212818 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212866 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212918 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.212986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.212945 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.313944 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.313907 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.313969 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.313998 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314035 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314283 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314192 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314283 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314265 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjpdv\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314323 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314344 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314376 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314469 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314484 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314668 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314533 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.314668 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.314601 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.315433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.315406 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.316650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.316617 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.316834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.316812 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.377617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.377529 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjpdv\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.377617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.377543 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.418509 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.418477 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:36.858471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.858432 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 17:51:36.893501 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:51:36.893468 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc574dffc_6350_48be_830c_d905dcbea1ea.slice/crio-4e836a2b42cbc9e5b45d3ff157199dbfa784f28eba507c004465abb8875a6a1e WatchSource:0}: Error finding container 4e836a2b42cbc9e5b45d3ff157199dbfa784f28eba507c004465abb8875a6a1e: Status 404 returned error can't find the container with id 4e836a2b42cbc9e5b45d3ff157199dbfa784f28eba507c004465abb8875a6a1e Nov 03 17:51:36.896013 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.895979 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:36.896100 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.896069 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:36.896100 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:36.896095 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:37.374397 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.374349 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" event={"ID":"c574dffc-6350-48be-830c-d905dcbea1ea","Type":"ContainerStarted","Data":"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e"} Nov 03 17:51:37.374910 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.374419 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" event={"ID":"c574dffc-6350-48be-830c-d905dcbea1ea","Type":"ContainerStarted","Data":"4e836a2b42cbc9e5b45d3ff157199dbfa784f28eba507c004465abb8875a6a1e"} Nov 03 17:51:37.426016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.425972 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:37.426219 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.426142 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:37.426286 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.426219 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:37.499804 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:37.499741 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podStartSLOduration=1.499724232 podStartE2EDuration="1.499724232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:37.498508833 +0000 UTC m=+1077.327592454" watchObservedRunningTime="2026-11-03 17:51:37.499724232 +0000 UTC m=+1077.328807833" Nov 03 17:51:38.419005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:38.418964 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:38.419431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:38.419037 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:39.418796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:39.418765 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:39.419001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:39.418831 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:40.419178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:40.419137 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:40.419585 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:40.419204 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:41.418964 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:41.418930 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:41.419178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:41.418990 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:42.418886 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:42.418846 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:42.419364 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:42.418905 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:43.418794 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:43.418764 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" start-of-body= Nov 03 17:51:43.418999 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:43.418823 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": dial tcp 10.132.0.60:15021: connect: connection refused" Nov 03 17:51:44.424063 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:44.424029 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:44.424492 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:44.424410 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:44.425076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:44.425054 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 17:51:44.519485 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:51:44.519448 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:51:50.657471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.657440 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 17:51:50.670117 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.670088 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.698772 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.698746 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.698932 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.698776 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95mbz\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.698932 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.698810 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.698943 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.698989 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.699012 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.699062 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.699116 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.699178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.699167 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.725084 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.725063 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-07f297-openshift-default-dockercfg-dfdjr\"" Nov 03 17:51:50.776400 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.776356 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 17:51:50.800136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800089 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800198 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800229 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95mbz\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800274 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800322 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800370 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800391 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800405 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.800553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800430 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.801201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800809 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.801201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.800907 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.801201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.801062 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.801201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.801152 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.801829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.801803 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.803093 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.803048 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.803226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.803210 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.866659 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.866632 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.866866 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.866843 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-95mbz\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:50.984282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:50.984250 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:51.426725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:51.426698 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 17:51:51.428951 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:51:51.428923 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf03b91bd_ab96_4797_ad4c_e214ba415a4a.slice/crio-ba7e2df6e227e12e6a392dba96f7d7c0c5ce9bb23b106997ccf142137688bbc6 WatchSource:0}: Error finding container ba7e2df6e227e12e6a392dba96f7d7c0c5ce9bb23b106997ccf142137688bbc6: Status 404 returned error can't find the container with id ba7e2df6e227e12e6a392dba96f7d7c0c5ce9bb23b106997ccf142137688bbc6 Nov 03 17:51:51.431839 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:51.431810 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:51.431938 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:51.431874 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:51.431938 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:51.431925 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:51.463724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:51.463695 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" event={"ID":"f03b91bd-ab96-4797-ad4c-e214ba415a4a","Type":"ContainerStarted","Data":"ba7e2df6e227e12e6a392dba96f7d7c0c5ce9bb23b106997ccf142137688bbc6"} Nov 03 17:51:52.485429 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:52.485391 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" event={"ID":"f03b91bd-ab96-4797-ad4c-e214ba415a4a","Type":"ContainerStarted","Data":"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4"} Nov 03 17:51:52.711191 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:52.711135 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podStartSLOduration=2.711120003 podStartE2EDuration="2.711120003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:52.659280366 +0000 UTC m=+1092.488363971" watchObservedRunningTime="2026-11-03 17:51:52.711120003 +0000 UTC m=+1092.540203606" Nov 03 17:51:52.984904 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:52.984869 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:52.986402 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:52.986367 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:52.986492 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:52.986435 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:53.480890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.480846 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 17:51:53.485417 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.485391 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533165 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533131 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533330 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533405 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533473 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533539 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5fxn\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533595 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533623 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533678 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.533789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.533714 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.540473 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.540453 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-548ca1-openshift-default-dockercfg-z6wdl\"" Nov 03 17:51:53.613260 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.605404 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 17:51:53.635278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635250 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635281 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635460 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635300 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635460 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635420 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635535 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635487 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635614 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635644 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635661 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635737 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5fxn\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635846 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635933 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635890 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.635976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.635930 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.636046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.636025 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.636727 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.636704 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.637996 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.637969 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.638211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.638192 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.699125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.699094 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5fxn\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.699125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.699122 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.800970 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.800853 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:53.985698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.985662 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:53.985902 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:53.985731 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:54.284790 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.284754 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:54.284929 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.284834 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:54.284929 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.284875 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:51:54.285224 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.285197 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 17:51:54.498482 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.498444 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" event={"ID":"2b85d0fd-a322-40fb-9cec-0da989e071ab","Type":"ContainerStarted","Data":"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4"} Nov 03 17:51:54.498670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.498506 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" event={"ID":"2b85d0fd-a322-40fb-9cec-0da989e071ab","Type":"ContainerStarted","Data":"dc7d4d4ca9c17204b80f1492f5b09ada12d291205db8109b21de8fc09022bddb"} Nov 03 17:51:54.801684 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:54.801649 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:51:55.012632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.004088 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:55.012632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.004151 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:55.806597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.805652 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:51:55.806597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.805717 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:51:56.004598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.999433 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:56.004598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:55.999494 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:56.806277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:56.805704 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:51:56.806277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:56.805784 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:51:56.987594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:56.987023 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:56.987594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:56.987098 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:57.801640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:57.801561 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:51:57.801860 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:57.801649 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:51:57.985152 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:57.985108 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body= Nov 03 17:51:57.985307 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:57.985177 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" Nov 03 17:51:58.801312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:58.801276 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:51:58.801767 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:58.801334 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:51:58.989395 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:58.989362 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:58.989667 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:58.989644 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:58.990417 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:58.990400 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 17:51:59.204585 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:59.204520 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podStartSLOduration=6.204504976 podStartE2EDuration="6.204504976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:51:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:51:54.623752198 +0000 UTC m=+1094.452835802" watchObservedRunningTime="2026-11-03 17:51:59.204504976 +0000 UTC m=+1099.033588580" Nov 03 17:51:59.801241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:59.801203 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:51:59.801415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:51:59.801272 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:52:00.802149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:00.802112 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body= Nov 03 17:52:00.802658 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:00.802173 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" Nov 03 17:52:01.807302 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:01.807253 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:52:01.807861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:01.807565 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:52:01.808431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:01.808402 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 17:52:06.938594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.938549 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 17:52:06.942245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.942229 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968168 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968323 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968202 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968323 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968230 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968414 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968331 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968414 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968357 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968483 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968644 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968603 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjtw\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968702 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968675 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.968758 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.968710 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:06.997107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:06.997086 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-42c162-openshift-default-dockercfg-jf8vf\"" Nov 03 17:52:07.049096 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.049070 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086122 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086222 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjtw\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086266 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086300 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086403 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086441 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086480 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086534 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.086557 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.087249 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.097115 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.101528 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.100899 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.102424 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.101839 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.102973 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.102950 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.107435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.106970 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.118998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.118968 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.160120 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.160052 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjtw\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.162734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.162710 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.255743 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.255676 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:07.688396 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.688363 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 17:52:07.689787 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:52:07.689762 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33122dc2_1320_4916_82bd_ea2a19add58b.slice/crio-4c8a29b94d96ffee674286fb7f1c2a7d55b6e59f49cd4a36818341b7ade30d5c WatchSource:0}: Error finding container 4c8a29b94d96ffee674286fb7f1c2a7d55b6e59f49cd4a36818341b7ade30d5c: Status 404 returned error can't find the container with id 4c8a29b94d96ffee674286fb7f1c2a7d55b6e59f49cd4a36818341b7ade30d5c Nov 03 17:52:07.692155 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.692122 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:07.692250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.692186 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:07.692250 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:07.692215 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:52:08.592480 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:08.592441 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" event={"ID":"33122dc2-1320-4916-82bd-ea2a19add58b","Type":"ContainerStarted","Data":"552db5f262098ac1ec52a2153304d21d4f4c93f972f9e55effd2066f7d7dbc6f"} Nov 03 17:52:08.592953 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:08.592509 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" event={"ID":"33122dc2-1320-4916-82bd-ea2a19add58b","Type":"ContainerStarted","Data":"4c8a29b94d96ffee674286fb7f1c2a7d55b6e59f49cd4a36818341b7ade30d5c"} Nov 03 17:52:08.780660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:08.780597 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podStartSLOduration=2.780557217 podStartE2EDuration="2.780557217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:08.779938579 +0000 UTC m=+1108.609022181" watchObservedRunningTime="2026-11-03 17:52:08.780557217 +0000 UTC m=+1108.609640819" Nov 03 17:52:09.274597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:09.271815 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:09.274597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:09.271879 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:09.274597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:09.272148 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:10.258977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:10.258938 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:10.259636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:10.259602 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:11.266605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:11.265946 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:11.266605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:11.266009 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:12.258003 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:12.257958 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:12.258300 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:12.258268 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:13.257187 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:13.257151 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:13.257842 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:13.257790 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:14.257129 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.257093 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body= Nov 03 17:52:14.257318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.257159 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" Nov 03 17:52:14.535943 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:52:14.535845 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:52:14.564977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.564944 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-p6r9n-wqvv2"] Nov 03 17:52:14.569165 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.569140 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.631341 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.631290 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p6r9n-wqvv2"] Nov 03 17:52:14.696001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.695959 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.696186 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.696079 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.696186 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.696126 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.696306 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.696227 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmcx6\" (UniqueName: \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.797356 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.797271 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.797356 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.797339 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.797545 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.797368 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.797545 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.797400 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmcx6\" (UniqueName: \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.797821 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.797798 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.798053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.798034 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.798495 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.798475 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.861288 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.861246 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmcx6\" (UniqueName: \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\") pod \"periodic-gathering-p6r9n-wqvv2\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:14.882001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:14.881971 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:52:15.260335 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.260313 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-p6r9n-wqvv2"] Nov 03 17:52:15.262663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.262637 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:15.263174 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:52:15.263146 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod936eb31a_d9c5_4e39_9585_5c3c7ce7103f.slice/crio-3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a WatchSource:0}: Error finding container 3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a: Status 404 returned error can't find the container with id 3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a Nov 03 17:52:15.263307 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.263187 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:15.264201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.264183 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 17:52:15.645838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.645739 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" event={"ID":"936eb31a-d9c5-4e39-9585-5c3c7ce7103f","Type":"ContainerStarted","Data":"3ec7a2e291c8c91797a0a56a14ed7abf4ed543fc20c1f858d12ea30c0dfd6af0"} Nov 03 17:52:15.646009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.645842 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" event={"ID":"936eb31a-d9c5-4e39-9585-5c3c7ce7103f","Type":"ContainerStarted","Data":"3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a"} Nov 03 17:52:15.771013 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:52:15.770961 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" podStartSLOduration=1.770946689 podStartE2EDuration="1.770946689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:52:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:52:15.767916969 +0000 UTC m=+1115.597000572" watchObservedRunningTime="2026-11-03 17:52:15.770946689 +0000 UTC m=+1115.600030287" Nov 03 17:52:44.547134 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:52:44.547101 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:53:14.533302 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:53:14.533259 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:53:22.031035 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.030997 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 17:53:22.035127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.035107 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.090188 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.090153 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"kube-root-ca.crt\"" Nov 03 17:53:22.140149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.140113 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"default-dockercfg-5nxkq\"" Nov 03 17:53:22.140325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.140213 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"openshift-service-ca.crt\"" Nov 03 17:53:22.141529 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.141505 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"distinct-w2-42c162-kserve-self-signed-certs\"" Nov 03 17:53:22.144412 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.144387 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 17:53:22.169861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.169821 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.170126 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.170105 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.170296 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.170277 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8nkc\" (UniqueName: \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.170540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.170521 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.170738 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.170721 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.170921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.170904 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272341 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272384 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272437 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8nkc\" (UniqueName: \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272520 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272544 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272617 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272884 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272803 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.272884 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.272872 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.273080 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.273053 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.275096 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.275069 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.275225 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.275209 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.337901 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.337823 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8nkc\" (UniqueName: \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\") pod \"distinct-w2-42c162-kserve-5c464d9f4f-brr6k\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.347350 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.347326 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:22.777969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:22.777935 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 17:53:22.779690 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:53:22.779664 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod02a2ef7b_f802_4ae4_9967_c99c1340c3db.slice/crio-aa26020d2796d4f7ad4aa4fa1700c91a00afafe11afdff037e246a433f726827 WatchSource:0}: Error finding container aa26020d2796d4f7ad4aa4fa1700c91a00afafe11afdff037e246a433f726827: Status 404 returned error can't find the container with id aa26020d2796d4f7ad4aa4fa1700c91a00afafe11afdff037e246a433f726827 Nov 03 17:53:23.083211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:23.083129 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" event={"ID":"02a2ef7b-f802-4ae4-9967-c99c1340c3db","Type":"ContainerStarted","Data":"97c74595d86eed842748a7ad0311b4a73b1ad5cd3ef5b1e29bedb117233303f4"} Nov 03 17:53:23.083211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:23.083168 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" event={"ID":"02a2ef7b-f802-4ae4-9967-c99c1340c3db","Type":"ContainerStarted","Data":"aa26020d2796d4f7ad4aa4fa1700c91a00afafe11afdff037e246a433f726827"} Nov 03 17:53:23.203463 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:23.203416 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podStartSLOduration=2.203402542 podStartE2EDuration="2.203402542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:23.202684957 +0000 UTC m=+1183.031768575" watchObservedRunningTime="2026-11-03 17:53:23.203402542 +0000 UTC m=+1183.032486176" Nov 03 17:53:23.347828 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:23.347741 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:32.347933 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:32.347895 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:32.360505 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:32.360485 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:33.149679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:33.149643 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:53:37.297172 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:37.297138 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:53:37.297633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:37.297346 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" containerID="cri-o://355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb" gracePeriod=30 Nov 03 17:53:40.448737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.448697 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:53:40.449246 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.448833 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:53:40.454299 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.454279 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:53:40.454563 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.454544 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:53:40.933361 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.933268 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 17:53:40.937449 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:40.937423 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.038772 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.038739 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"openshift-service-ca.crt\"" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044426 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4q2b\" (UniqueName: \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044483 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044518 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044641 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044677 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044743 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.044938 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"kube-root-ca.crt\"" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.045075 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"default-dockercfg-czx7h\"" Nov 03 17:53:41.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.045183 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-73a606\"/\"distinct-w4-73a606-kserve-self-signed-certs\"" Nov 03 17:53:41.149258 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.149219 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.149522 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.149505 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.149756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.149739 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.149890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.149877 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.150076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.150061 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.150228 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.150213 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4q2b\" (UniqueName: \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.183829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.152843 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.183829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.153052 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.183829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.153236 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.183829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.167435 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 17:53:41.188912 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.188830 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.207634 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.189448 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.223636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.223607 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4q2b\" (UniqueName: \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\") pod \"distinct-w4-73a606-kserve-7c4d5df658-mgdc8\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.252883 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.252854 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:41.847345 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:41.847322 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 17:53:41.850283 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:53:41.850249 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06c2c4c2_285a_4490_9a50_b653f4f808d5.slice/crio-36a042de23733a3fc346328012bb16ee882448e0366b23d4e98dabcfb4cbbcf5 WatchSource:0}: Error finding container 36a042de23733a3fc346328012bb16ee882448e0366b23d4e98dabcfb4cbbcf5: Status 404 returned error can't find the container with id 36a042de23733a3fc346328012bb16ee882448e0366b23d4e98dabcfb4cbbcf5 Nov 03 17:53:42.225426 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.225385 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" event={"ID":"06c2c4c2-285a-4490-9a50-b653f4f808d5","Type":"ContainerStarted","Data":"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11"} Nov 03 17:53:42.225639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.225433 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" event={"ID":"06c2c4c2-285a-4490-9a50-b653f4f808d5","Type":"ContainerStarted","Data":"36a042de23733a3fc346328012bb16ee882448e0366b23d4e98dabcfb4cbbcf5"} Nov 03 17:53:42.253502 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.253469 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:42.593956 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.593926 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:53:42.704249 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704217 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704293 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704331 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704382 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704469 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704537 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704548 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704529 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704616 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704667 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704720 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\") pod \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\" (UID: \"136e96f2-2bd9-4fed-9bc7-152b9bb9827d\") " Nov 03 17:53:42.704983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704802 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:42.704983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.704903 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:42.705149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705132 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:42.705231 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705212 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.705270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705234 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.705270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705245 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.705270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705255 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.705270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.705263 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.707338 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.707314 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 17:53:42.707462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.707390 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "kube-api-access-t8rml". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:42.707462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.707420 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:42.707462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.707413 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token" pod "136e96f2-2bd9-4fed-9bc7-152b9bb9827d" (UID: "136e96f2-2bd9-4fed-9bc7-152b9bb9827d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:42.730982 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.730936 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podStartSLOduration=2.730922009 podStartE2EDuration="2.730922009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:53:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:53:42.357422304 +0000 UTC m=+1202.186505916" watchObservedRunningTime="2026-11-03 17:53:42.730922009 +0000 UTC m=+1202.560005611" Nov 03 17:53:42.807083 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.807047 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.807083 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.807081 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.807273 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.807096 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8rml\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-kube-api-access-t8rml\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:42.807273 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:42.807112 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/136e96f2-2bd9-4fed-9bc7-152b9bb9827d-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:43.231806 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.231775 4980 generic.go:356] "Generic (PLEG): container finished" podID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerID="355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb" exitCode=0 Nov 03 17:53:43.232285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.231821 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" Nov 03 17:53:43.232285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.231857 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" event={"ID":"136e96f2-2bd9-4fed-9bc7-152b9bb9827d","Type":"ContainerDied","Data":"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb"} Nov 03 17:53:43.232285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.231911 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4" event={"ID":"136e96f2-2bd9-4fed-9bc7-152b9bb9827d","Type":"ContainerDied","Data":"87bcd2ff0b43afad0229a3cbc63a68f7978963d4c776556a752ea87804b5f72d"} Nov 03 17:53:43.232285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.231935 4980 scope.go:122] "RemoveContainer" containerID="355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb" Nov 03 17:53:43.242288 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.242267 4980 scope.go:122] "RemoveContainer" containerID="355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb" Nov 03 17:53:43.242564 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:53:43.242542 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb\": container with ID starting with 355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb not found: ID does not exist" containerID="355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb" Nov 03 17:53:43.242632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.242589 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb"} err="failed to get container status \"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb\": rpc error: code = NotFound desc = could not find container \"355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb\": container with ID starting with 355f8860114a4188564a3e28e3cfd6f1f0747215f25d40510826c243138134cb not found: ID does not exist" Nov 03 17:53:43.453751 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.453718 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:53:43.504983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:43.504904 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-qs9s4"] Nov 03 17:53:44.292759 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:44.292724 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" path="/var/lib/kubelet/pods/136e96f2-2bd9-4fed-9bc7-152b9bb9827d/volumes" Nov 03 17:53:44.521484 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:53:44.521447 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:53:51.254523 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:51.254436 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:51.267404 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:51.267383 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:51.288290 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:51.288267 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:53:52.284026 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:52.283994 4980 generic.go:356] "Generic (PLEG): container finished" podID="936eb31a-d9c5-4e39-9585-5c3c7ce7103f" containerID="3ec7a2e291c8c91797a0a56a14ed7abf4ed543fc20c1f858d12ea30c0dfd6af0" exitCode=0 Nov 03 17:53:52.284554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:52.284072 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" event={"ID":"936eb31a-d9c5-4e39-9585-5c3c7ce7103f","Type":"ContainerDied","Data":"3ec7a2e291c8c91797a0a56a14ed7abf4ed543fc20c1f858d12ea30c0dfd6af0"} Nov 03 17:53:53.436374 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.436350 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:53:53.527230 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527198 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\") pod \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " Nov 03 17:53:53.527381 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527251 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\") pod \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " Nov 03 17:53:53.527381 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527315 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\" (UniqueName: \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\") pod \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " Nov 03 17:53:53.527473 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527423 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\") pod \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\" (UID: \"936eb31a-d9c5-4e39-9585-5c3c7ce7103f\") " Nov 03 17:53:53.527608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527529 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path" pod "936eb31a-d9c5-4e39-9585-5c3c7ce7103f" (UID: "936eb31a-d9c5-4e39-9585-5c3c7ce7103f"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 17:53:53.527796 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527677 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle" pod "936eb31a-d9c5-4e39-9585-5c3c7ce7103f" (UID: "936eb31a-d9c5-4e39-9585-5c3c7ce7103f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:53.527850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527823 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle" pod "936eb31a-d9c5-4e39-9585-5c3c7ce7103f" (UID: "936eb31a-d9c5-4e39-9585-5c3c7ce7103f"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 17:53:53.527892 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527855 4980 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-archives-path\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:53.527892 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.527871 4980 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-trusted-ca-bundle\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:53.529881 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.529856 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6" pod "936eb31a-d9c5-4e39-9585-5c3c7ce7103f" (UID: "936eb31a-d9c5-4e39-9585-5c3c7ce7103f"). InnerVolumeSpecName "kube-api-access-mmcx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 17:53:53.629401 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.629323 4980 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-service-ca-bundle\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:53.629401 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:53.629354 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmcx6\" (UniqueName: \"kubernetes.io/projected/936eb31a-d9c5-4e39-9585-5c3c7ce7103f-kube-api-access-mmcx6\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 17:53:54.302833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:54.302792 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" Nov 03 17:53:54.303010 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:54.302791 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-p6r9n-wqvv2" event={"ID":"936eb31a-d9c5-4e39-9585-5c3c7ce7103f","Type":"ContainerDied","Data":"3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a"} Nov 03 17:53:54.303010 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:53:54.302915 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3a6012782550b8fb268251f61b1010780e74454e7fd628f09aa5431ba88fca8a" Nov 03 17:54:07.014584 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.014543 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 17:54:07.015082 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015073 4980 cpu_manager.go:414] "containerMap: removing container" podUID="936eb31a-d9c5-4e39-9585-5c3c7ce7103f" containerName="insights-gathering" Nov 03 17:54:07.015153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015085 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="936eb31a-d9c5-4e39-9585-5c3c7ce7103f" containerName="insights-gathering" Nov 03 17:54:07.015153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015107 4980 cpu_manager.go:414] "containerMap: removing container" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" Nov 03 17:54:07.015153 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015112 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" Nov 03 17:54:07.015315 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015179 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="936eb31a-d9c5-4e39-9585-5c3c7ce7103f" containerName="insights-gathering" Nov 03 17:54:07.015315 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.015191 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="136e96f2-2bd9-4fed-9bc7-152b9bb9827d" containerName="istio-proxy" Nov 03 17:54:07.018560 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.018541 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084296 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084263 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084347 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084367 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084559 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084496 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgch\" (UniqueName: \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084559 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084538 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.084738 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.084719 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.122206 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.122181 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-c76280\"/\"distinct-2-w1-c76280-kserve-self-signed-certs\"" Nov 03 17:54:07.124738 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.124714 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 17:54:07.185843 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.185803 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186032 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.185919 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186032 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.185988 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186032 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186011 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186202 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186069 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgch\" (UniqueName: \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186202 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186102 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186354 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186705 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186608 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.186791 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.186769 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.188470 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.188440 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.188600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.188464 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.249819 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.249795 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgch\" (UniqueName: \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\") pod \"distinct-2-w1-c76280-kserve-659fff668f-j9wqk\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.330943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.330858 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:07.345761 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.345735 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 17:54:07.349667 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.349643 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388388 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388440 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388484 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388511 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388543 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.388607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388587 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.389041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388713 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.389041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388760 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6fkl\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.389041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.388801 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.460536 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.441834 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491010 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491054 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491090 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491115 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491251 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491293 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6fkl\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491328 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491447 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491478 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.491928 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.492543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.492177 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.494026 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.493993 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.494141 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.494018 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.494529 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.494508 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.494645 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.494626 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.495128 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.495094 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.557155 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.557127 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.557506 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.557483 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6fkl\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.676690 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.676616 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:07.749658 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:54:07.749622 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2eb99fa0_9a7e_4a6a_be3b_80043d5a3b59.slice/crio-53b1ead063da0016579bc9728fef23002c1c041add7360ea41e12976a853e002 WatchSource:0}: Error finding container 53b1ead063da0016579bc9728fef23002c1c041add7360ea41e12976a853e002: Status 404 returned error can't find the container with id 53b1ead063da0016579bc9728fef23002c1c041add7360ea41e12976a853e002 Nov 03 17:54:07.749834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:07.749808 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 17:54:08.060713 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.060689 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 17:54:08.062646 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:54:08.062618 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17572545_3b6d_4d65_83a9_968c02a960c9.slice/crio-033eec56a6615d781bbab7fb0056db1d45b62fa84cea57bfb60acca5dfaf88c4 WatchSource:0}: Error finding container 033eec56a6615d781bbab7fb0056db1d45b62fa84cea57bfb60acca5dfaf88c4: Status 404 returned error can't find the container with id 033eec56a6615d781bbab7fb0056db1d45b62fa84cea57bfb60acca5dfaf88c4 Nov 03 17:54:08.065227 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.065201 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:54:08.065309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.065260 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:54:08.065309 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.065290 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:54:08.386049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.385947 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" event={"ID":"17572545-3b6d-4d65-83a9-968c02a960c9","Type":"ContainerStarted","Data":"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504"} Nov 03 17:54:08.386049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.386017 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" event={"ID":"17572545-3b6d-4d65-83a9-968c02a960c9","Type":"ContainerStarted","Data":"033eec56a6615d781bbab7fb0056db1d45b62fa84cea57bfb60acca5dfaf88c4"} Nov 03 17:54:08.388356 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.388329 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" event={"ID":"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59","Type":"ContainerStarted","Data":"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da"} Nov 03 17:54:08.388510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.388366 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" event={"ID":"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59","Type":"ContainerStarted","Data":"53b1ead063da0016579bc9728fef23002c1c041add7360ea41e12976a853e002"} Nov 03 17:54:08.511776 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.511722 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podStartSLOduration=1.511707505 podStartE2EDuration="1.511707505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:08.510108592 +0000 UTC m=+1228.339192196" watchObservedRunningTime="2026-11-03 17:54:08.511707505 +0000 UTC m=+1228.340791104" Nov 03 17:54:08.636503 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.636399 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podStartSLOduration=2.636377761 podStartE2EDuration="2.636377761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:08.636001722 +0000 UTC m=+1228.465085325" watchObservedRunningTime="2026-11-03 17:54:08.636377761 +0000 UTC m=+1228.465461366" Nov 03 17:54:08.679721 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:08.679676 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:09.331653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:09.331623 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:09.681471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:09.681387 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:09.681471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:09.681444 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:10.678000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:10.677960 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:10.678651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:10.678618 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:11.681219 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:11.681181 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:11.681764 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:11.681334 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:12.680114 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:12.680067 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:12.680314 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:12.680122 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:13.679011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:13.678971 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:13.679677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:13.679644 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:14.519826 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:54:14.519784 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:54:14.677488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:14.677450 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" start-of-body= Nov 03 17:54:14.677656 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:14.677505 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": dial tcp 10.132.0.68:15021: connect: connection refused" Nov 03 17:54:15.681886 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:15.681856 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:15.682332 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:15.682300 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:15.683042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:15.683021 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 17:54:17.331673 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:17.331646 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:17.343925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:17.343899 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:17.452265 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:17.452242 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:54:44.520307 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:54:44.520268 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:54:48.092056 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.092024 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 17:54:48.097451 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.097419 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.151998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.151967 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"kube-root-ca.crt\"" Nov 03 17:54:48.152149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.152009 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"openshift-service-ca.crt\"" Nov 03 17:54:48.175087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175052 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.175241 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175092 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.175286 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175258 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.175333 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175319 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd6sp\" (UniqueName: \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.175441 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175416 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.175489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.175464 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.200825 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.200800 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"default-dockercfg-dn566\"" Nov 03 17:54:48.200957 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.200893 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-07f297\"/\"distinct-2-w0-07f297-kserve-self-signed-certs\"" Nov 03 17:54:48.204278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.204257 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.276612 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.276740 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.276772 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.276897 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.276976 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd6sp\" (UniqueName: \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.277353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.277008 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.278238 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.278209 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.278544 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.278516 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.278653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.278529 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.282171 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.281891 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.282533 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.282509 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.344164 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.344085 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd6sp\" (UniqueName: \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\") pod \"distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.410532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.410502 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:48.832832 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:48.832806 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 17:54:48.834371 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:54:48.834347 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc296cbf0_a68b_41db_b467_57a28081f9f1.slice/crio-07f6da192503675047075bce48162c1a00b4ecb1c299e3fd5479f07aa557a5b3 WatchSource:0}: Error finding container 07f6da192503675047075bce48162c1a00b4ecb1c299e3fd5479f07aa557a5b3: Status 404 returned error can't find the container with id 07f6da192503675047075bce48162c1a00b4ecb1c299e3fd5479f07aa557a5b3 Nov 03 17:54:49.640048 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:49.640012 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" event={"ID":"c296cbf0-a68b-41db-b467-57a28081f9f1","Type":"ContainerStarted","Data":"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978"} Nov 03 17:54:49.640543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:49.640057 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" event={"ID":"c296cbf0-a68b-41db-b467-57a28081f9f1","Type":"ContainerStarted","Data":"07f6da192503675047075bce48162c1a00b4ecb1c299e3fd5479f07aa557a5b3"} Nov 03 17:54:49.760489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:49.760438 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podStartSLOduration=1.760423718 podStartE2EDuration="1.760423718s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:54:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:54:49.75925667 +0000 UTC m=+1269.588340273" watchObservedRunningTime="2026-11-03 17:54:49.760423718 +0000 UTC m=+1269.589507320" Nov 03 17:54:50.410942 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:50.410907 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:58.411498 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:58.411464 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:58.424786 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:58.424759 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:54:58.700205 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:54:58.700178 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:55:14.520389 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:55:14.520348 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:55:37.672182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.672143 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 17:55:37.676007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.675989 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.784160 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.784119 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 17:55:37.800152 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800116 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800332 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800251 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800332 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800302 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800460 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sztf\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800497 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800522 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800644 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800681 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.800816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.800770 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902161 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902121 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902161 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902166 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902399 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902228 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sztf\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902399 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902247 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902399 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902264 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902399 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902294 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902399 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902321 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902687 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902435 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902687 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902471 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902820 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902793 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902906 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902883 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.902980 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.902957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.903067 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.903040 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.903187 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.903132 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.905081 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.905059 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.905292 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.905270 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.970634 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.970601 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.970785 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.970660 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sztf\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:37.990625 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:37.990555 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:38.460481 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.460451 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 17:55:38.462047 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:55:38.462019 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode692a971_c058_4d22_8bdd_0992fef3c254.slice/crio-b0d170c39acac96b6ad00daaee91bef950406a123bffdbaf36637f2e4c7ef5a7 WatchSource:0}: Error finding container b0d170c39acac96b6ad00daaee91bef950406a123bffdbaf36637f2e4c7ef5a7: Status 404 returned error can't find the container with id b0d170c39acac96b6ad00daaee91bef950406a123bffdbaf36637f2e4c7ef5a7 Nov 03 17:55:38.464439 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.464407 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:38.464515 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.464472 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:38.464515 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.464509 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:55:38.918052 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.917967 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" event={"ID":"e692a971-c058-4d22-8bdd-0992fef3c254","Type":"ContainerStarted","Data":"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904"} Nov 03 17:55:38.918052 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.918033 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" event={"ID":"e692a971-c058-4d22-8bdd-0992fef3c254","Type":"ContainerStarted","Data":"b0d170c39acac96b6ad00daaee91bef950406a123bffdbaf36637f2e4c7ef5a7"} Nov 03 17:55:38.990679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:38.990648 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:39.042749 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:39.042691 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podStartSLOduration=2.042671127 podStartE2EDuration="2.042671127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:39.041807885 +0000 UTC m=+1318.870891487" watchObservedRunningTime="2026-11-03 17:55:39.042671127 +0000 UTC m=+1318.871754731" Nov 03 17:55:39.992159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:39.992121 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:39.992632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:39.992179 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:40.991950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:40.991916 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:40.992176 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:40.991973 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:41.996640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:41.991289 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:41.996640 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:41.991352 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:42.993046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:42.993004 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:42.993323 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:42.993293 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:43.997531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:43.997495 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:43.998206 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:43.998172 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:44.650826 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:55:44.650782 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:55:44.991199 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:44.991157 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" start-of-body= Nov 03 17:55:44.991371 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:44.991231 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": dial tcp 10.132.0.70:15021: connect: connection refused" Nov 03 17:55:46.412083 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:46.412049 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:46.412621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:46.412423 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:46.413937 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:46.413911 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 17:55:53.823819 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.823782 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 17:55:53.828034 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.828015 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.902783 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.902742 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.902962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.902801 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.902962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.902884 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.902962 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.902920 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfvgh\" (UniqueName: \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.903106 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.903066 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.903215 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.903202 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:53.931285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.931244 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-42c162\"/\"distinct-2-w2-42c162-kserve-self-signed-certs\"" Nov 03 17:55:53.934063 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:53.934037 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 17:55:54.004875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.004829 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.005201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.005182 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.005339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.005321 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.005475 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.005441 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.005661 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.005482 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfvgh\" (UniqueName: \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.005661 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.005589 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.007354 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.007293 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.008534 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.008452 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.010219 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.008807 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.011141 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.011118 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.015553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.015525 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.071135 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.071099 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfvgh\" (UniqueName: \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\") pod \"distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.139703 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.139616 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:54.576873 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:54.576846 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 17:55:54.578400 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:55:54.578375 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1c3f6539_35b7_484e_ad83_c27292358b62.slice/crio-55cdfb26909fa0c0de5dd54f974240a2337d6cfd9445b1061f73a2aaf198fbe7 WatchSource:0}: Error finding container 55cdfb26909fa0c0de5dd54f974240a2337d6cfd9445b1061f73a2aaf198fbe7: Status 404 returned error can't find the container with id 55cdfb26909fa0c0de5dd54f974240a2337d6cfd9445b1061f73a2aaf198fbe7 Nov 03 17:55:55.069402 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:55.069366 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" event={"ID":"1c3f6539-35b7-484e-ad83-c27292358b62","Type":"ContainerStarted","Data":"249dc588dafd433aa7bb84220b2e40470c473da460e17c829e554d0d8dde2888"} Nov 03 17:55:55.069837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:55.069418 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" event={"ID":"1c3f6539-35b7-484e-ad83-c27292358b62","Type":"ContainerStarted","Data":"55cdfb26909fa0c0de5dd54f974240a2337d6cfd9445b1061f73a2aaf198fbe7"} Nov 03 17:55:55.139812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:55.139781 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:55:55.192007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:55:55.191947 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podStartSLOduration=2.191931248 podStartE2EDuration="2.191931248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:55:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:55:55.189314665 +0000 UTC m=+1335.018398269" watchObservedRunningTime="2026-11-03 17:55:55.191931248 +0000 UTC m=+1335.021014852" Nov 03 17:56:04.140137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:04.140062 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:56:04.153777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:04.153748 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:56:05.143065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:05.143034 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:56:08.027916 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.027872 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 17:56:08.032045 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.032022 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084503 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084545 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084608 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmmx7\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084698 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084738 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084800 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084863 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.084976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.084894 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.085122 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.085012 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186552 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186668 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186690 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186718 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmmx7\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186734 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186751 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.186818 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186770 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186829 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.186853 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.187028 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.187102 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.187202 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187495 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.187475 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.187621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.187601 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.189464 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.189444 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.189554 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.189485 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.257377 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.257309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmmx7\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.258055 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.258017 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 17:56:08.259193 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.259162 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.346716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.346634 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:08.781835 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.781807 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 17:56:08.783758 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:56:08.783728 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5829e122_cc6e_4a48_9468_868a1eefdf37.slice/crio-163870aae496072f24c1bda622b1132d0223afebd343cf57611bc12cc58f0802 WatchSource:0}: Error finding container 163870aae496072f24c1bda622b1132d0223afebd343cf57611bc12cc58f0802: Status 404 returned error can't find the container with id 163870aae496072f24c1bda622b1132d0223afebd343cf57611bc12cc58f0802 Nov 03 17:56:08.786620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.786566 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:08.786716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.786659 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:08.786716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:08.786691 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:09.186157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.186047 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" event={"ID":"5829e122-cc6e-4a48-9468-868a1eefdf37","Type":"ContainerStarted","Data":"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349"} Nov 03 17:56:09.186157 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.186114 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" event={"ID":"5829e122-cc6e-4a48-9468-868a1eefdf37","Type":"ContainerStarted","Data":"163870aae496072f24c1bda622b1132d0223afebd343cf57611bc12cc58f0802"} Nov 03 17:56:09.347009 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.346973 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:09.352591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.348908 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:09.352591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.348964 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:09.372663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:09.368524 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podStartSLOduration=2.36850017 podStartE2EDuration="2.36850017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:09.361270347 +0000 UTC m=+1349.190353955" watchObservedRunningTime="2026-11-03 17:56:09.36850017 +0000 UTC m=+1349.197583773" Nov 03 17:56:10.347164 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:10.347128 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:10.347753 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:10.347187 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:11.347598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:11.347216 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:11.347598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:11.347339 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:12.347566 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:12.347526 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:12.347782 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:12.347638 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:13.347825 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:13.347787 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:13.348282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:13.347855 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:14.347959 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:14.347928 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:14.348405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:14.347979 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:14.519825 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:56:14.519787 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:56:15.347400 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:15.347364 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" start-of-body= Nov 03 17:56:15.347654 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:15.347434 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.72:15021/healthz/ready\": dial tcp 10.132.0.72:15021: connect: connection refused" Nov 03 17:56:16.512791 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:16.512748 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:16.513262 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:16.513081 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:16.514072 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:16.514038 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 17:56:39.118864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.118831 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 17:56:39.123024 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.122999 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166510 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166474 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166529 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166662 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166768 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166726 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166806 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166767 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166872 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rlmd\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.166967 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166955 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.167004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.166990 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.167037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.167023 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.230207 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.230175 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 17:56:39.267805 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267769 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.267975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267817 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.267975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267843 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.267975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267898 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.267975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267925 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.267975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.267973 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268002 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268036 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268113 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rlmd\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268237 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268267 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268507 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268640 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.268880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.268691 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.271360 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.271340 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.271677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.271656 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.331394 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.331358 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.331637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.331618 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rlmd\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.437074 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.437001 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:39.894247 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.894214 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 17:56:39.897867 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.897835 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:39.897987 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.897901 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:39.897987 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:39.897929 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:40.487519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:40.487470 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" event={"ID":"468f327b-7814-4a45-b798-33ff5139b58e","Type":"ContainerStarted","Data":"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83"} Nov 03 17:56:40.488014 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:40.487537 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" event={"ID":"468f327b-7814-4a45-b798-33ff5139b58e","Type":"ContainerStarted","Data":"1d0adf3519a043abb2cbb44c5dbb4ce3e8f66a22641c47077bf2b76a090abe98"} Nov 03 17:56:40.614589 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:40.614530 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podStartSLOduration=1.614516356 podStartE2EDuration="1.614516356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:40.611111771 +0000 UTC m=+1380.440195372" watchObservedRunningTime="2026-11-03 17:56:40.614516356 +0000 UTC m=+1380.443599958" Nov 03 17:56:41.438155 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:41.438122 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:41.439758 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:41.439733 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:41.439873 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:41.439783 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:42.437479 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:42.437448 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:42.437994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:42.437510 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:43.438669 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:43.438633 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:43.439073 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:43.438683 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:44.437852 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:44.437816 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:44.438028 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:44.437882 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:44.524040 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:56:44.524005 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:56:45.437301 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:45.437265 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:45.437501 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:45.437329 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:46.437551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:46.437516 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" start-of-body= Nov 03 17:56:46.437963 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:46.437597 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": dial tcp 10.132.0.73:15021: connect: connection refused" Nov 03 17:56:47.503118 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:47.503074 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:47.503812 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:47.503788 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:47.505276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:47.505255 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 17:56:54.092656 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.092218 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 17:56:54.127792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.123913 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.266592 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.175291 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.552326 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.625399 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638148 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638220 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638377 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638542 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqgw2\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638632 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638672 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638700 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638779 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.654599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.638989 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:54.802635 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.771240 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:56:54.857598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.851703 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:56:54.857598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.852257 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:56:54.857598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.852555 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:56:54.956598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.941244 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.128801 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129034 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129063 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqgw2\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129126 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129159 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129193 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129293 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.130606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.129346 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.149620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.143494 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.149620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:54.955079 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.149620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.144150 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.149620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.084718 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:55.150998 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.150967 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.151841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.151816 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.152133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.152098 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:55.152621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.152597 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:56:55.153080 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.153054 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:56:55.276070 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.257339 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.299600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.294943 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.345976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.345846 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.367345 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.367309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqgw2\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:55.538162 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:55.538064 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:56.021203 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.021166 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 17:56:56.033526 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.033484 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.085431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.085388 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 17:56:56.093945 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:56:56.093913 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf828a59b_451c_4fea_a029_c8a3af694295.slice/crio-4b47e0ede04a133c7ffd4768c381b8e8424985032278297eac0bb6b0c5688f80 WatchSource:0}: Error finding container 4b47e0ede04a133c7ffd4768c381b8e8424985032278297eac0bb6b0c5688f80: Status 404 returned error can't find the container with id 4b47e0ede04a133c7ffd4768c381b8e8424985032278297eac0bb6b0c5688f80 Nov 03 17:56:56.104614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.094734 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 17:56:56.104614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.095966 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:56.104614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.099833 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:56.104614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.099915 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:56.135237 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135203 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135733 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135273 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135733 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135406 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135733 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135512 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135733 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135694 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cswc8\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135853 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135926 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.135981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.135972 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.136132 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.136087 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237208 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237265 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237332 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237373 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237413 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cswc8\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237498 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237555 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237619 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237667 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237720 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237788 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237764 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.237822 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.237795 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.238208 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.238181 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.238646 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.238187 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.240195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.240170 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.240411 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.240388 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.323346 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.323131 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.336639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.331947 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cswc8\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.342046 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.342010 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" event={"ID":"f828a59b-451c-4fea-a029-c8a3af694295","Type":"ContainerStarted","Data":"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0"} Nov 03 17:56:56.342215 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.342070 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" event={"ID":"f828a59b-451c-4fea-a029-c8a3af694295","Type":"ContainerStarted","Data":"4b47e0ede04a133c7ffd4768c381b8e8424985032278297eac0bb6b0c5688f80"} Nov 03 17:56:56.349259 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.349232 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:56.476964 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.476892 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podStartSLOduration=2.476872864 podStartE2EDuration="2.476872864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:56.471401506 +0000 UTC m=+1396.300485109" watchObservedRunningTime="2026-11-03 17:56:56.476872864 +0000 UTC m=+1396.305956466" Nov 03 17:56:56.539147 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.539106 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:56:56.790519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.790487 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 17:56:56.792603 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:56:56.792531 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode948d145_ce64_4ecc_b176_8f2c5b65d9e6.slice/crio-462effef535a88530959620dcc5fb8a7917dcccf5ab825fcb1f45514b0813458 WatchSource:0}: Error finding container 462effef535a88530959620dcc5fb8a7917dcccf5ab825fcb1f45514b0813458: Status 404 returned error can't find the container with id 462effef535a88530959620dcc5fb8a7917dcccf5ab825fcb1f45514b0813458 Nov 03 17:56:56.795306 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.795278 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:56.795384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.795354 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:56.795431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:56.795385 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:56:57.350383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:57.350332 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" event={"ID":"e948d145-ce64-4ecc-b176-8f2c5b65d9e6","Type":"ContainerStarted","Data":"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69"} Nov 03 17:56:57.350981 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:57.350427 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" event={"ID":"e948d145-ce64-4ecc-b176-8f2c5b65d9e6","Type":"ContainerStarted","Data":"462effef535a88530959620dcc5fb8a7917dcccf5ab825fcb1f45514b0813458"} Nov 03 17:56:57.541003 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:57.540953 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:56:57.541170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:57.541016 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:56:58.349977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:58.349937 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:56:58.351584 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:58.351539 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:56:58.351933 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:58.351609 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:56:58.539019 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:58.538984 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:56:58.539197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:58.539044 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:56:59.350290 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:59.350251 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:56:59.350483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:59.350309 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:56:59.539624 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:59.539586 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:56:59.540122 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:56:59.539653 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:57:00.350513 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:00.350479 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:57:00.350730 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:00.350531 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:57:00.539054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:00.538994 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:57:00.539326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:00.539290 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:57:01.350696 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:01.350658 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:57:01.351190 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:01.350726 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:57:01.539136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:01.539012 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:57:01.539136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:01.539064 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:57:02.349798 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:02.349760 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:57:02.350261 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:02.350237 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:57:02.539698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:02.539657 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body= Nov 03 17:57:02.540165 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:02.539732 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" Nov 03 17:57:03.350304 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:03.350268 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body= Nov 03 17:57:03.350502 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:03.350334 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" Nov 03 17:57:03.966503 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:03.966465 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:57:03.967508 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:03.967481 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:57:03.968317 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:03.968298 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 17:57:04.195793 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:04.195729 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podStartSLOduration=9.19571187 podStartE2EDuration="9.19571187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:56:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:56:57.480940349 +0000 UTC m=+1397.310023951" watchObservedRunningTime="2026-11-03 17:57:04.19571187 +0000 UTC m=+1404.024795499" Nov 03 17:57:04.567907 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:04.567849 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:57:04.568407 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:04.568377 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:57:04.569672 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:04.569646 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 17:57:08.460507 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.460477 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 17:57:08.464597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.464561 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.511472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.511440 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 17:57:08.515685 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.515666 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.563564 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.563534 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 17:57:08.581961 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.581931 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 17:57:08.586184 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.586163 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.604934 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.604907 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605072 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.604945 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605072 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.604983 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605189 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605102 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605228 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605186 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605262 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605220 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605311 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605295 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605355 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605340 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605446 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605432 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605478 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605462 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d24v\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605520 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605505 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605540 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lcsvs\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605586 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605618 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.605651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605642 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605751 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605687 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605797 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605751 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.605863 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.605840 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.628798 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.628767 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 17:57:08.654844 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.654811 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 17:57:08.707193 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707160 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.707193 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707191 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.707436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707234 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.707436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707310 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.707436 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707338 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.707661 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707630 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.707850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707826 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.707926 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707865 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.707926 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707894 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.708037 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707962 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.708144 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708066 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.708233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.707894 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.708384 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708360 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.708515 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708483 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708522 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708550 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708617 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708678 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708708 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708784 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708811 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708839 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708867 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7wd\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708876 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708913 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708944 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d24v\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708952 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.708977 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709080 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709115 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709148 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lcsvs\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709178 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709251 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709317 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709520 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.709837 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.709628 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.710735 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.710305 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.710735 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.710426 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.713322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.712695 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.713322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.713080 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.713946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.713921 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.770043 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.770010 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.771681 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.771652 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.772001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.771984 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lcsvs\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.772189 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.772167 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d24v\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.780123 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.780094 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:08.810595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810544 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.810744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810635 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.810744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810660 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7wd\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.810744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810697 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.810923 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810795 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.810986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810927 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.811042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.810970 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.811042 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811016 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.811150 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811075 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.811208 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811183 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.811328 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811308 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.812289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811484 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.812289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.811521 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.812477 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.812453 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.813753 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.813727 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.814326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.814304 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.829392 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.829352 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:08.858936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.858899 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 17:57:08.877793 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.877758 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:08.887605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.887326 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.887605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.887536 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7wd\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.909720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.909682 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:08.933606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:08.933461 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 17:57:09.015597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015500 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015588 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015623 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015653 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015685 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015814 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.015931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015877 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.016312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.015989 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nfhx\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.016312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.016030 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.117489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.117449 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nfhx\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.117729 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.117509 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118073 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118049 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118146 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118110 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118263 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118242 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118298 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118338 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118369 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118535 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118516 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118607 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.118788 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118768 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.119005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.118986 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.119285 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.119256 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.119385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.119265 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.120920 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.120899 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.127005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.121946 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.172051 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.172020 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 17:57:09.172524 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:57:09.172493 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda01e1041_e6a8_46a1_99bf_8f5f622df595.slice/crio-761e20a18d8e74acf3d2040c34ff70f51723d7788103f177cbafea389a8741c8 WatchSource:0}: Error finding container 761e20a18d8e74acf3d2040c34ff70f51723d7788103f177cbafea389a8741c8: Status 404 returned error can't find the container with id 761e20a18d8e74acf3d2040c34ff70f51723d7788103f177cbafea389a8741c8 Nov 03 17:57:09.175332 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.175294 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.175419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.175379 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.175458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.175416 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.180871 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.180847 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nfhx\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.181000 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.180979 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\") pod \"e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.191907 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.191856 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 17:57:09.197525 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.197505 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.205342 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.205320 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:09.263729 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.263638 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 17:57:09.312629 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:57:09.312556 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ada2ca0_4a3a_4b35_b97f_a13ac669f3ee.slice/crio-7c4ed8a0c38782834fb6051c7285e607818d1d5c50c3dcde940cd63e0e6ea363 WatchSource:0}: Error finding container 7c4ed8a0c38782834fb6051c7285e607818d1d5c50c3dcde940cd63e0e6ea363: Status 404 returned error can't find the container with id 7c4ed8a0c38782834fb6051c7285e607818d1d5c50c3dcde940cd63e0e6ea363 Nov 03 17:57:09.314924 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.313678 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 17:57:09.315599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.315532 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.315700 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.315635 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.315700 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.315679 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.329378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329311 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329361 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prqn2\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329545 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329453 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329545 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329519 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329560 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329605 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329671 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329862 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329755 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.329862 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.329799 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.372312 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.372076 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 17:57:09.377331 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:57:09.376406 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2d53fa_70fe_4707_a617_6b6ccc87321c.slice/crio-065c0ba90798029d9582909a39b6de83412e03961072eec815dc3e877a82f72f WatchSource:0}: Error finding container 065c0ba90798029d9582909a39b6de83412e03961072eec815dc3e877a82f72f: Status 404 returned error can't find the container with id 065c0ba90798029d9582909a39b6de83412e03961072eec815dc3e877a82f72f Nov 03 17:57:09.378744 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.378711 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.378855 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.378791 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.378855 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.378834 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.432861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.432833 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.432899 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.432926 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433131 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433006 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433243 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433131 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433305 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433289 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433394 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433381 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433507 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433411 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prqn2\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433620 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433561 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.433960 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433938 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.434276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.434245 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.434832 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.433223 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.435529 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.435503 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.435856 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.435834 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.438031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.437500 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.438031 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.437942 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.477497 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.477418 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" event={"ID":"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee","Type":"ContainerStarted","Data":"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b"} Nov 03 17:57:09.477497 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.477491 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" event={"ID":"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee","Type":"ContainerStarted","Data":"7c4ed8a0c38782834fb6051c7285e607818d1d5c50c3dcde940cd63e0e6ea363"} Nov 03 17:57:09.485927 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.485674 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" event={"ID":"3c2d53fa-70fe-4707-a617-6b6ccc87321c","Type":"ContainerStarted","Data":"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914"} Nov 03 17:57:09.485927 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.485743 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" event={"ID":"3c2d53fa-70fe-4707-a617-6b6ccc87321c","Type":"ContainerStarted","Data":"065c0ba90798029d9582909a39b6de83412e03961072eec815dc3e877a82f72f"} Nov 03 17:57:09.488514 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.488479 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" event={"ID":"a01e1041-e6a8-46a1-99bf-8f5f622df595","Type":"ContainerStarted","Data":"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f"} Nov 03 17:57:09.488702 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.488537 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" event={"ID":"a01e1041-e6a8-46a1-99bf-8f5f622df595","Type":"ContainerStarted","Data":"761e20a18d8e74acf3d2040c34ff70f51723d7788103f177cbafea389a8741c8"} Nov 03 17:57:09.497409 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.497381 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.501857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.498195 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-prqn2\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.523610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.521842 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:09.613805 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.609888 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podStartSLOduration=1.6098668250000001 podStartE2EDuration="1.609866825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:09.60236636 +0000 UTC m=+1409.431449955" watchObservedRunningTime="2026-11-03 17:57:09.609866825 +0000 UTC m=+1409.438950427" Nov 03 17:57:09.620029 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.618783 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 17:57:09.643215 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:57:09.643171 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbc7243_1aa6_4c57_be08_00a2988e99c1.slice/crio-9121c590828df6c750618c655ed39716391508dd6ec02ecf6e76e7c4cb90ccf5 WatchSource:0}: Error finding container 9121c590828df6c750618c655ed39716391508dd6ec02ecf6e76e7c4cb90ccf5: Status 404 returned error can't find the container with id 9121c590828df6c750618c655ed39716391508dd6ec02ecf6e76e7c4cb90ccf5 Nov 03 17:57:09.649509 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.648439 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.649509 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.648531 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.649509 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.648596 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:09.752315 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.752231 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podStartSLOduration=1.752193712 podStartE2EDuration="1.752193712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:09.749640886 +0000 UTC m=+1409.578724480" watchObservedRunningTime="2026-11-03 17:57:09.752193712 +0000 UTC m=+1409.581277316" Nov 03 17:57:09.783041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.780325 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:09.829751 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.829655 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:09.877745 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.877126 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podStartSLOduration=1.877106637 podStartE2EDuration="1.877106637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:09.874469938 +0000 UTC m=+1409.703553533" watchObservedRunningTime="2026-11-03 17:57:09.877106637 +0000 UTC m=+1409.706190242" Nov 03 17:57:09.912217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.912130 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:09.952329 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:09.952295 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 17:57:10.005289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.004327 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:10.005289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.004425 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:10.005289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.004468 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:10.617650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.568307 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" event={"ID":"6df1d58a-ceab-439e-8fa7-7496a990fa97","Type":"ContainerStarted","Data":"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c"} Nov 03 17:57:10.617650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.568407 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" event={"ID":"6df1d58a-ceab-439e-8fa7-7496a990fa97","Type":"ContainerStarted","Data":"9a7cda1e23d658b1485abd9897c17dd04f66d238f87781b5d13ab3f11e591663"} Nov 03 17:57:10.617650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.597996 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" event={"ID":"dcbc7243-1aa6-4c57-be08-00a2988e99c1","Type":"ContainerStarted","Data":"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945"} Nov 03 17:57:10.617650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.598079 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" event={"ID":"dcbc7243-1aa6-4c57-be08-00a2988e99c1","Type":"ContainerStarted","Data":"9121c590828df6c750618c655ed39716391508dd6ec02ecf6e76e7c4cb90ccf5"} Nov 03 17:57:10.722597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.722024 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podStartSLOduration=1.722001777 podStartE2EDuration="1.722001777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:10.717016789 +0000 UTC m=+1410.546100392" watchObservedRunningTime="2026-11-03 17:57:10.722001777 +0000 UTC m=+1410.551085380" Nov 03 17:57:10.853059 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.853018 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:10.864670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.858972 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:10.864670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.859671 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:10.864670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.859718 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:10.915878 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.915789 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:10.916125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.916096 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:10.928079 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:10.918497 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podStartSLOduration=2.91847572 podStartE2EDuration="2.91847572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:10.885513779 +0000 UTC m=+1410.714597382" watchObservedRunningTime="2026-11-03 17:57:10.91847572 +0000 UTC m=+1410.747559326" Nov 03 17:57:11.221882 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.221844 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:11.222325 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.222295 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:11.222529 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.222503 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:11.393272 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.393222 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:11.487647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.487378 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:11.524368 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.524334 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:11.524713 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.524690 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:11.524913 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.524888 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:11.603324 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.601415 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:11.630709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.630659 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:11.712777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.712727 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:11.785076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.784981 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:11.785336 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.785297 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:11.833629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.833558 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:11.833922 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.833887 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:11.914596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.913011 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:11.914596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:11.913079 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:12.089734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.081741 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.162666 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.150654 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.227714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.227675 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:12.228029 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.227996 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:12.294596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.289658 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.411674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.411543 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.412262 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.412236 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:57:12.492477 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.487927 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.492477 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.488316 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:12.527609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.526231 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:12.527609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.526298 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:12.656601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.652442 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.664798 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.806289 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.806353 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.723720 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.808520 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.818184 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.818228 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.848233 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:12.974362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:12.848293 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:13.184657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.127760 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.184657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.128208 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:57:13.220600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.194942 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:13.220600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.195003 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:13.224953 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.224867 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:13.225171 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.225145 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:13.287600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.271130 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.287600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.271155 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.287600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.271389 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:13.358799 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.358734 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.420614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.420553 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.488852 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.488272 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.583877 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.583828 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:13.584211 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.584162 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:13.811358 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.810721 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:13.823675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.811947 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:13.823675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.812196 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:13.823675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.815709 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:13.823675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.815770 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:13.852596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.842846 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:13.852596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.842909 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:13.859139 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.859099 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:13.933277 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.933232 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.004544 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:13.936635 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:14.146848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.128534 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:14.218598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.212417 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.218598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.212475 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:14.250596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.248128 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 03 17:57:14.271550 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.271507 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:14.423653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.422942 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:14.459440 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.459388 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:14.492759 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.492716 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:14.547327 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.547290 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.547671 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.547638 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:14.927600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.919852 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.927600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.919912 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:14.939759 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.939715 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:14.943636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.941048 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:14.944065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.941077 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:14.944424 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.941313 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.944584 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.944542 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:14.944703 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.942638 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:14.944815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:14.944795 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:15.134317 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:57:15.134283 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:57:15.357734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.357689 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:57:15.358395 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.358348 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.414652 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.434059 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.572072 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.527248 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.481695 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:15.577679 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.540347 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:15.611706 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.611610 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:15.611969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.611937 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:15.791201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.791151 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:15.791517 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.791474 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:15.858829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.858774 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:15.902654 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.861715 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:15.910280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.910244 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.079695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.036664 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:16.079695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.971107 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.079695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.935698 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:16.079695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.971142 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:16.079695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:15.971164 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.209604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.205704 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.209604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.205766 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:16.361596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.361369 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.547790 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.547696 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.548090 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.548049 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:16.591606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.577607 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.591606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.589654 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:16.592816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.592775 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.632757 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.632706 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:16.705596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.702505 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:57:16.782643 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.782207 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.782901 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.782868 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:16.831280 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.831177 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.833603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.833514 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:16.944598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.919545 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:16.944598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:16.919622 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:17.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.051167 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.051497 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.051791 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:17.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.052032 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:17.066597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.052325 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:57:17.135659 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.099263 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": context deadline exceeded" Nov 03 17:57:17.223599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.223245 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:17.223599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.223306 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:17.390656 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.376264 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.527719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.527684 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:17.527991 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.527949 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:17.583773 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.583705 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.593774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.593717 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.638817 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.638742 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:17.659001 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.658891 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:17.805599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.799562 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:17.805599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.799650 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:17.848599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.837622 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:17.848599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.837699 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:17.916691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.915961 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:17.916691 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:17.916027 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:18.052935 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.052876 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.053675 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.053641 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.054133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.054111 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:18.055617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.055593 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.215013 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.214976 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:18.222741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.222695 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:18.343602 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.334604 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.381251 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.378288 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.528660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.527967 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:18.528660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.528030 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:18.593634 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.591706 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.609598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.596646 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.645594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.644968 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:18.664718 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.661773 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:18.761142 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.761097 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:57:18.763742 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.763050 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:57:18.782210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.781998 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" start-of-body= Nov 03 17:57:18.782210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.782066 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.76:15021/healthz/ready\": dial tcp 10.132.0.76:15021: connect: connection refused" Nov 03 17:57:18.833847 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.832891 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" start-of-body= Nov 03 17:57:18.833847 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.832955 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": dial tcp 10.132.0.77:15021: connect: connection refused" Nov 03 17:57:18.910024 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.909992 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:18.910304 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.910276 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:18.997598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:18.974861 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:19.046692 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.045843 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:57:19.088040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.075408 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.088040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.075711 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:19.281065 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.281028 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:19.281684 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.281650 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:19.428789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.418023 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.428789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.420754 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.526716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.526661 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.582670 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.582631 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:19.621950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.617394 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.622276 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.622243 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:19.686212 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.686130 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:57:19.735751 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.734680 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:19.965671 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.965637 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" start-of-body= Nov 03 17:57:19.965943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:19.965909 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.78:15021/healthz/ready\": dial tcp 10.132.0.78:15021: connect: connection refused" Nov 03 17:57:20.116057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.116012 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:20.116756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.116725 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:20.230861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.230766 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" start-of-body= Nov 03 17:57:20.231045 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.230858 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.79:15021/healthz/ready\": dial tcp 10.132.0.79:15021: connect: connection refused" Nov 03 17:57:20.691607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.691552 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:20.724221 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.723889 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:20.728266 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.728191 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:20.805941 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.805906 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:57:20.806289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.806272 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:20.807331 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.807314 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:57:20.809316 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.809301 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:20.809728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.809713 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:57:20.815799 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.815777 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 17:57:20.880615 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.880009 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:20.880615 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.880068 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:20.909055 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.909023 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:20.941199 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.941159 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:20.991534 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:20.991457 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 17:57:21.134382 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.134325 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:21.140916 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.137113 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:21.141107 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.141018 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 17:57:21.344524 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.343209 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:21.344524 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.344317 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:21.345927 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.345879 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 17:57:21.526596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.525764 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" start-of-body= Nov 03 17:57:21.526596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:21.525822 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.80:15021/healthz/ready\": dial tcp 10.132.0.80:15021: connect: connection refused" Nov 03 17:57:22.527210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:22.527180 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:22.527704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:22.527471 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:22.528102 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:22.528084 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 17:57:28.161628 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:28.158976 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:28.303599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:28.290934 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:28.813877 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:28.813813 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:28.835649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:28.820106 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:28.962724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:28.962669 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.137657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.133931 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.161090 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.161046 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:29.162206 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.162181 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:57:29.332600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.295905 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.355600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.333228 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:29.441475 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.437717 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.442415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.442375 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.602608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.602297 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.625809 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.625752 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:29.671115 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.671060 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.882660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.833757 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.882660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.834203 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:57:29.882660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.857994 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:29.882660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.858436 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:57:29.966868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.966801 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:29.970800 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:29.970752 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:57:30.136776 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.136670 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:30.177600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.141036 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:57:30.178629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.168660 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:30.379601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.335689 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:30.379601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.360758 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:30.379601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.360832 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:30.472778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.472735 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:57:30.631589 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.631525 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:30.687855 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.684482 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:30.726653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.700341 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:57:30.785599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.780857 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:30.843606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.804011 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:57:30.849786 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.844002 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:30.862806 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.862734 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:30.971816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:30.971751 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:31.192598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.188994 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:31.265599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.192754 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:31.419861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.357834 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:31.530756 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.530655 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:31.576596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.576548 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:31.752770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.751226 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:31.884136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.884042 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:31.944757 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.906075 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:31.986797 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:31.986729 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.110594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.110221 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:32.110594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.110282 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.305600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.289331 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.541870 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.299713 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.648005 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.647901 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.707235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.299776 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:32.723595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.555315 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.773264 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.772362 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:32.774880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.500140 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:32.903631 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.903504 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:32.963596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.934846 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:32.984598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:32.950683 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:33.005182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.004891 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:33.277633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.004976 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:33.560652 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:57:33.535402 4980 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.21s" Nov 03 17:57:34.107092 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.078669 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:57:34.118133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.680604 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.119137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.680656 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:34.123965 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.123942 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:34.124207 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.124194 4980 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Nov 03 17:57:34.125260 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.877450 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:57:34.126504 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.877377 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.127289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.877413 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:57:34.127925 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.950078 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:34.135061 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:33.561306 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.135204 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.051667 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.135545 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.078271 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.300317 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.300262 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-f46zb" podUID="00efe73b-9c70-4536-9b6f-cbd8e4ca188d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.43:8000/ready\": context deadline exceeded" Nov 03 17:57:34.311690 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.311649 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.491652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.491626 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:34.491864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.491847 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:57:34.491967 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.491887 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:57:34.494398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.494375 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:34.502949 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.502926 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:57:34.521248 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.521227 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:57:34.523091 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.523060 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:34.526953 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.526930 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:57:34.532195 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:34.530350 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:57:40.105274 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.105244 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 17:57:40.114752 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.114735 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.174619 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.174584 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 17:57:40.216531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216501 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.216708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216559 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.216766 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216741 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.216821 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216787 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.216821 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216805 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.216969 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.216946 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.217057 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.217043 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.217117 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.217098 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.217163 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.217150 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8swdf\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.317777 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317742 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.317965 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317838 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.317965 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317867 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.317965 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317882 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.317965 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317915 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.317972 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318025 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318102 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8swdf\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318169 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318281 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318195 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318406 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318460 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318518 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.318932 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.318747 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.320441 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.320417 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.320543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.320451 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.381378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.381295 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.381724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.381702 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8swdf\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.428498 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.428471 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:40.870120 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.870079 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 17:57:40.870788 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:57:40.870761 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c69b51_d406_418a_9716_b418ac458773.slice/crio-636bdf0118078e9b9d03c6a1e234e1ed6cf49bc37e125ffd66ebd15afbc239c2 WatchSource:0}: Error finding container 636bdf0118078e9b9d03c6a1e234e1ed6cf49bc37e125ffd66ebd15afbc239c2: Status 404 returned error can't find the container with id 636bdf0118078e9b9d03c6a1e234e1ed6cf49bc37e125ffd66ebd15afbc239c2 Nov 03 17:57:40.873200 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.873169 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:40.873269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.873237 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:40.873269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:40.873264 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:57:41.621529 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:41.621485 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" event={"ID":"27c69b51-d406-418a-9716-b418ac458773","Type":"ContainerStarted","Data":"74b5e3886e83ca52e6b5e2c119071f2755b86a5ee59b13dcb29ece4d6efaf9f8"} Nov 03 17:57:41.622053 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:41.621547 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" event={"ID":"27c69b51-d406-418a-9716-b418ac458773","Type":"ContainerStarted","Data":"636bdf0118078e9b9d03c6a1e234e1ed6cf49bc37e125ffd66ebd15afbc239c2"} Nov 03 17:57:41.824719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:41.824660 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podStartSLOduration=1.824645504 podStartE2EDuration="1.824645504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:57:41.796113411 +0000 UTC m=+1441.625197013" watchObservedRunningTime="2026-11-03 17:57:41.824645504 +0000 UTC m=+1441.653729105" Nov 03 17:57:42.429594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:42.429544 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:42.431493 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:42.431468 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:42.431627 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:42.431514 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:43.429264 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:43.429223 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:43.429705 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:43.429297 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:44.429358 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:44.429321 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:44.429789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:44.429387 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:44.520890 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:57:44.520849 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:57:45.428834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:45.428793 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:45.429027 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:45.428872 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:46.429918 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:46.429737 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:46.429918 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:46.429818 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:47.429485 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:47.429440 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 17:57:47.429696 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:47.429508 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 17:57:48.502741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:48.502682 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:48.503961 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:48.503944 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:48.504850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:48.504827 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 17:57:53.054601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.051675 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.190536 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.190486 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.339771 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.333085 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.393618 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.391972 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:53.393618 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.392288 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.456941 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.359070 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.583782 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.583727 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:53.584499 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.584451 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.798668 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.767002 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:53.973931 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:53.973879 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.091098 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.090980 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:54.144191 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.144158 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:57:54.194040 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.193994 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:54.194467 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.194445 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:57:54.254536 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.254489 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.640597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.589714 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.640597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.590176 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.640597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.590639 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.640597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.591310 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.640597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.592696 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:54.643986 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.643945 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:54.678202 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.678168 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:57:54.754632 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.754539 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:57:54.754829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.754674 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:54.754829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.754719 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:57:54.754829 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.754756 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:57:54.916664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.905743 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:57:55.040525 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.040486 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:57:55.088604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:54.974911 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:55.092806 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.092777 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:57:55.211600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.207816 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:55.270740 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.270699 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:55.293466 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.293420 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 17:57:55.610654 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.594736 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:55.610654 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.609636 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:55.662605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.627126 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:55.662605 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.618654 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:55.669932 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.666743 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:55.719596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.715786 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:57:55.719596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:55.715851 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.47:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.056598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.054104 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.095638 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.091207 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.251209 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.224647 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:57:56.302598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.274810 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:56.597865 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.597072 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.622746 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.622695 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.654608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.641397 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.654608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.642814 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:56.684024 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:56.683961 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:57.086594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.084309 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:57.110598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.109850 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:57.110598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.109984 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:57.247608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.229406 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:57.302609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.302483 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:57.617607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.614816 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:57.627594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.623654 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:57:57.685840 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.684843 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:57.685840 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.685223 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:57.737599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:57.725733 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.097611 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.092688 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.127262 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.127205 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:58.275955 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.275912 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.477598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.473712 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": context deadline exceeded" Nov 03 17:57:58.495698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.495656 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:57:58.629595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.626343 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.629595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.626664 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:58.687435 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.687396 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:57:58.688026 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.687996 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:57:58.732784 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:58.732658 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:57:59.179656 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.179134 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:57:59.220133 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.220080 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:57:59.291597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.290755 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.501864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.501813 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.534428 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.519644 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.626786 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.626742 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.648921 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.648829 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.859652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.840547 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:57:59.859652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.840860 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:57:59.912976 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:57:59.912208 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:00.186490 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.186378 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:00.262657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.262601 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:00.313593 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.285641 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:58:00.313593 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.285746 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:58:00.313593 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.301703 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:58:00.584595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.528318 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:58:00.584977 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.584955 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:58:00.585104 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.585094 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:58:00.585208 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.585199 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:58:00.920666 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.917231 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:58:00.926636 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:00.924166 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:58:03.099563 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.099523 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 17:58:03.104650 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.104625 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.182314 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.182277 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 17:58:03.215972 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.215946 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-1c1ea78c-openshift-default-dockercfg-zsrx7\"" Nov 03 17:58:03.318613 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.267662 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.343924 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.324826 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.346543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.346511 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.346794 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.346777 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.346951 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.346937 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.347123 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.347044 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.347431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.347413 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.347590 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.347555 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4s7\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.347741 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.347725 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.448881 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.448849 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449038 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.448898 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449066 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449135 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449119 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449215 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449150 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449215 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449195 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449280 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449326 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449300 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4s7\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449442 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449419 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449500 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449453 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449544 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449643 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.449757 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.449712 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.450181 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.450160 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.451861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.451836 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.451943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.451857 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.545607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.545222 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.552842 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.552809 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4s7\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\") pod \"e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:03.724610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:03.724525 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:04.165354 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.165317 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 17:58:04.165911 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:04.165888 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0199e352_db8a_4f9f_a88e_1edcbf8eb738.slice/crio-3132ea5df2b93f6ff0ea708df6269245af7c63c2f0922af70776100bbfac83bc WatchSource:0}: Error finding container 3132ea5df2b93f6ff0ea708df6269245af7c63c2f0922af70776100bbfac83bc: Status 404 returned error can't find the container with id 3132ea5df2b93f6ff0ea708df6269245af7c63c2f0922af70776100bbfac83bc Nov 03 17:58:04.168313 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.168282 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:04.168403 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.168348 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:04.168403 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.168377 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:04.413278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.413181 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerStarted","Data":"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba"} Nov 03 17:58:04.413278 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.413243 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerStarted","Data":"3132ea5df2b93f6ff0ea708df6269245af7c63c2f0922af70776100bbfac83bc"} Nov 03 17:58:04.539182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.539125 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podStartSLOduration=1.539107123 podStartE2EDuration="1.539107123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:04.537387599 +0000 UTC m=+1464.366471202" watchObservedRunningTime="2026-11-03 17:58:04.539107123 +0000 UTC m=+1464.368190728" Nov 03 17:58:04.732600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:04.728704 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:05.730783 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:05.730755 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:06.426170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:06.426143 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:06.427041 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:06.427023 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:58:09.721531 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.721494 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 17:58:09.726719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.726701 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.775789 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.775753 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.775990 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.775959 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776049 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776000 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776128 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776109 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776166 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776145 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776202 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776173 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776239 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776206 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776322 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776300 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkxng\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.776362 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.776350 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.837734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.837691 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 17:58:09.877774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.877742 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.877774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.877785 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkxng\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878029 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.877815 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878029 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.877887 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878029 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878021 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878244 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878059 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878244 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878104 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878244 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878143 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878244 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878194 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878526 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878300 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878526 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878356 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878551 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.878689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.878666 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.879286 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.879261 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.881419 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.881395 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.881979 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.881952 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.951594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.939732 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 17:58:09.951594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.948936 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.955774 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.955708 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.955936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.955889 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkxng\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\") pod \"e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985274 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985347 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985494 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985555 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985623 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985674 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985712 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5bf\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985788 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:09.986652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:09.985863 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.007677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.000448 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 17:58:10.043339 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.043060 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:10.104054 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104015 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104230 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104147 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104230 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104199 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104347 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104319 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104402 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104345 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104459 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104405 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104512 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104458 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.104512 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.104483 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.115859 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.115824 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.116004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.115897 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5bf\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.116004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.116000 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.116781 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.116750 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.116781 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.116764 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.117170 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.117132 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.121137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.121109 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.123539 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.123494 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.181289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.181260 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.182684 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.182659 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5bf\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\") pod \"e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.339360 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.339279 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:10.529386 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.529356 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 17:58:10.530542 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:10.530511 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4a64170_0c8b_4f86_ba3a_e84320fa0472.slice/crio-0f5fcd3a62b2f691b9a7fc255d1d3e1a6634357df957424edb4176b285a2758a WatchSource:0}: Error finding container 0f5fcd3a62b2f691b9a7fc255d1d3e1a6634357df957424edb4176b285a2758a: Status 404 returned error can't find the container with id 0f5fcd3a62b2f691b9a7fc255d1d3e1a6634357df957424edb4176b285a2758a Nov 03 17:58:10.532987 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.532951 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:10.533101 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.533031 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:10.533101 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.533075 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:10.737019 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:10.736990 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd56a4cbc_cebf_435c_96ed_7950973ea815.slice/crio-ae4d1ca7f301549d8ac2238074e764a872903ac8938d51f353a2047548e02d1c WatchSource:0}: Error finding container ae4d1ca7f301549d8ac2238074e764a872903ac8938d51f353a2047548e02d1c: Status 404 returned error can't find the container with id ae4d1ca7f301549d8ac2238074e764a872903ac8938d51f353a2047548e02d1c Nov 03 17:58:10.737704 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.737614 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 17:58:10.739945 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.739910 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:10.740021 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.740005 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:10.740060 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:10.740048 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:11.503596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.503467 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" event={"ID":"d56a4cbc-cebf-435c-96ed-7950973ea815","Type":"ContainerStarted","Data":"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba"} Nov 03 17:58:11.503596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.503540 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" event={"ID":"d56a4cbc-cebf-435c-96ed-7950973ea815","Type":"ContainerStarted","Data":"ae4d1ca7f301549d8ac2238074e764a872903ac8938d51f353a2047548e02d1c"} Nov 03 17:58:11.507595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.507103 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" event={"ID":"b4a64170-0c8b-4f86-ba3a-e84320fa0472","Type":"ContainerStarted","Data":"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb"} Nov 03 17:58:11.507595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.507166 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" event={"ID":"b4a64170-0c8b-4f86-ba3a-e84320fa0472","Type":"ContainerStarted","Data":"0f5fcd3a62b2f691b9a7fc255d1d3e1a6634357df957424edb4176b285a2758a"} Nov 03 17:58:11.743372 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.741146 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podStartSLOduration=2.741122286 podStartE2EDuration="2.741122286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.688501478 +0000 UTC m=+1471.517585081" watchObservedRunningTime="2026-11-03 17:58:11.741122286 +0000 UTC m=+1471.570205897" Nov 03 17:58:11.816101 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:11.816006 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podStartSLOduration=2.81599245 podStartE2EDuration="2.81599245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:11.812050225 +0000 UTC m=+1471.641133828" watchObservedRunningTime="2026-11-03 17:58:11.81599245 +0000 UTC m=+1471.645076051" Nov 03 17:58:12.044592 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.044531 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:12.045890 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.045860 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:12.046027 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.045914 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:12.340365 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.340271 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:12.341615 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.341584 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:12.341748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:12.341647 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:13.044724 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:13.044689 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:13.045113 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:13.044753 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:13.340621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:13.340514 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:13.340621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:13.340605 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:14.044888 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:14.044848 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:14.045318 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:14.044906 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:14.340708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:14.340624 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:14.340862 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:14.340701 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:14.539382 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:58:14.539262 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:58:15.044601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:15.044544 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:15.044783 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:15.044628 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:15.340549 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:15.340474 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:15.340994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:15.340973 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:16.050405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.050364 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:16.050749 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.050718 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:16.172282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.162228 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 17:58:16.377947 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.377384 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:16.377947 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.377450 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:16.560596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.541282 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.630095 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.611016 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-c74b2c26-dockercfg-kt5bp\"" Nov 03 17:58:16.630095 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.611833 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-c74b2c26\"" Nov 03 17:58:16.649597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.634628 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.650251 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.650214 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wp2nf\" (UniqueName: \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.768599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.764522 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wp2nf\" (UniqueName: \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.777900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.777861 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.787706 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.787672 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.847603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.847493 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 17:58:16.847603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.847590 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 17:58:16.897646 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.896760 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wp2nf\" (UniqueName: \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\") pod \"payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:16.901596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.900013 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 17:58:16.901596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.900132 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:16.907189 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:16.906043 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:17.009600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.007641 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.009600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.007791 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zdvq\" (UniqueName: \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.044553 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.044524 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:17.044712 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.044650 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:17.119603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.109383 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.119603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.109519 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zdvq\" (UniqueName: \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.126658 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.120342 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.190460 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.190426 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zdvq\" (UniqueName: \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\") pod \"payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.317343 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.317235 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:17.340084 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.340006 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:17.340257 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.340115 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:17.381967 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.381518 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 17:58:17.391270 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:17.391242 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ed52d39_d4ce_4301_a087_a3e2e43e68a1.slice/crio-e2e83141d022d7f7d1a7043bb7b6457e24b906b5e5de1ab62c61321ca2044666 WatchSource:0}: Error finding container e2e83141d022d7f7d1a7043bb7b6457e24b906b5e5de1ab62c61321ca2044666: Status 404 returned error can't find the container with id e2e83141d022d7f7d1a7043bb7b6457e24b906b5e5de1ab62c61321ca2044666 Nov 03 17:58:17.733880 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.733513 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 17:58:17.737021 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:17.736987 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a381410_0dab_449b_bcfc_b45b2b924329.slice/crio-000a38472eff3c014a3835b5e49ab5a014637da36b1d2515d409376912a567e3 WatchSource:0}: Error finding container 000a38472eff3c014a3835b5e49ab5a014637da36b1d2515d409376912a567e3: Status 404 returned error can't find the container with id 000a38472eff3c014a3835b5e49ab5a014637da36b1d2515d409376912a567e3 Nov 03 17:58:17.755228 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.754722 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" event={"ID":"6ed52d39-d4ce-4301-a087-a3e2e43e68a1","Type":"ContainerStarted","Data":"ff845c7d12f843da871b0b7d9167ea5f7318eff92f789857e9406b596d894c7e"} Nov 03 17:58:17.755366 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.755286 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" event={"ID":"6ed52d39-d4ce-4301-a087-a3e2e43e68a1","Type":"ContainerStarted","Data":"e2e83141d022d7f7d1a7043bb7b6457e24b906b5e5de1ab62c61321ca2044666"} Nov 03 17:58:17.755366 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.755342 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:17.756719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.756690 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" event={"ID":"6a381410-0dab-449b-bcfc-b45b2b924329","Type":"ContainerStarted","Data":"000a38472eff3c014a3835b5e49ab5a014637da36b1d2515d409376912a567e3"} Nov 03 17:58:17.882221 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:17.882152 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" podStartSLOduration=1.882132192 podStartE2EDuration="1.882132192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:17.873699353 +0000 UTC m=+1477.702782956" watchObservedRunningTime="2026-11-03 17:58:17.882132192 +0000 UTC m=+1477.711215796" Nov 03 17:58:18.050651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.048137 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" start-of-body= Nov 03 17:58:18.050651 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.048198 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.83:15021/healthz/ready\": dial tcp 10.132.0.83:15021: connect: connection refused" Nov 03 17:58:18.340076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.339975 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" start-of-body= Nov 03 17:58:18.340076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.340046 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.84:15021/healthz/ready\": dial tcp 10.132.0.84:15021: connect: connection refused" Nov 03 17:58:18.774621 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.773212 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" event={"ID":"6a381410-0dab-449b-bcfc-b45b2b924329","Type":"ContainerStarted","Data":"4c73e5aa0cc25f94346ab59764a8b716fb38fbb26e316c1131290760d1719da3"} Nov 03 17:58:18.899488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:18.899320 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" podStartSLOduration=2.899300004 podStartE2EDuration="2.899300004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:18.89323873 +0000 UTC m=+1478.722322331" watchObservedRunningTime="2026-11-03 17:58:18.899300004 +0000 UTC m=+1478.728383606" Nov 03 17:58:19.087648 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.086848 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:19.105595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.093260 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:19.109591 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.106854 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 17:58:19.352707 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.352626 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:19.353088 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.353054 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:19.355932 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.353985 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 17:58:19.782452 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:19.782402 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:23.756604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.756519 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 17:58:23.766856 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.763655 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:23.854610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.853779 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 17:58:23.880594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.872887 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-1c1ea78c\"" Nov 03 17:58:23.880594 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.873042 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-1c1ea78c-dockercfg-j5v68\"" Nov 03 17:58:23.927834 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.927745 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5492\" (UniqueName: \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:23.927999 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.927930 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:23.929519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.929486 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 17:58:23.943652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:23.943022 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.044649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.029031 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5492\" (UniqueName: \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:24.044649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.029170 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:24.044649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.030358 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:24.140642 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.134631 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.140642 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.134761 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbsg\" (UniqueName: \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.239247 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.239167 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.239390 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.239263 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbsg\" (UniqueName: \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.243193 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.240344 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.243382 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.243349 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 17:58:24.292945 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.292919 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5492\" (UniqueName: \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\") pod \"payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:24.311661 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.311557 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbsg\" (UniqueName: \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\") pod \"payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.436120 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.436086 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:24.548348 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.548317 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:24.821411 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:24.821380 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2fc39aa_5d93_4cb4_aac2_90b0c9fc601a.slice/crio-295bcaab4624b8edf0743eea2069a1466ec846f6fc931f8cbf8ce9c940b20000 WatchSource:0}: Error finding container 295bcaab4624b8edf0743eea2069a1466ec846f6fc931f8cbf8ce9c940b20000: Status 404 returned error can't find the container with id 295bcaab4624b8edf0743eea2069a1466ec846f6fc931f8cbf8ce9c940b20000 Nov 03 17:58:24.821795 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.821416 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 17:58:24.845432 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.845402 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" event={"ID":"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a","Type":"ContainerStarted","Data":"295bcaab4624b8edf0743eea2069a1466ec846f6fc931f8cbf8ce9c940b20000"} Nov 03 17:58:24.935080 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:24.935050 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 17:58:24.935616 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:24.935565 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb202ae_af49_4b2d_b612_c1fce977126d.slice/crio-f786db901d3aea97ebd2b02bbbfe22380fe484d385553b17fc98477c5418d341 WatchSource:0}: Error finding container f786db901d3aea97ebd2b02bbbfe22380fe484d385553b17fc98477c5418d341: Status 404 returned error can't find the container with id f786db901d3aea97ebd2b02bbbfe22380fe484d385553b17fc98477c5418d341 Nov 03 17:58:25.854201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.854169 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" event={"ID":"8fb202ae-af49-4b2d-b612-c1fce977126d","Type":"ContainerStarted","Data":"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5"} Nov 03 17:58:25.854677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.854233 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:25.854677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.854262 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" event={"ID":"8fb202ae-af49-4b2d-b612-c1fce977126d","Type":"ContainerStarted","Data":"f786db901d3aea97ebd2b02bbbfe22380fe484d385553b17fc98477c5418d341"} Nov 03 17:58:25.856018 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.855993 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" event={"ID":"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a","Type":"ContainerStarted","Data":"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065"} Nov 03 17:58:25.856124 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.856036 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:25.974540 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:25.974487 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" podStartSLOduration=2.974472687 podStartE2EDuration="2.974472687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:25.973208064 +0000 UTC m=+1485.802291680" watchObservedRunningTime="2026-11-03 17:58:25.974472687 +0000 UTC m=+1485.803556283" Nov 03 17:58:26.113677 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:26.108820 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" podStartSLOduration=3.108800577 podStartE2EDuration="3.108800577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:26.103882867 +0000 UTC m=+1485.932966471" watchObservedRunningTime="2026-11-03 17:58:26.108800577 +0000 UTC m=+1485.937884180" Nov 03 17:58:28.293601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.285517 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.338943 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.338834 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.607817 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.607712 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.626792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.626742 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.697171 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.689680 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.697830 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.697455 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:28.798916 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:28.798878 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 17:58:29.161623 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.129518 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:58:29.173007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.132703 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.221604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.221532 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.305378 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.305304 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:29.314035 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.314000 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:58:29.414654 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.387632 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.453601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.452858 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.537598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.512653 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 03 17:58:29.649599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.614302 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.649599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.614749 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:58:29.697660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.662695 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:58:29.697660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.663223 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:58:29.697660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.694302 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:29.697660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.694781 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:58:29.706076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.706040 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:58:29.771259 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:29.771226 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:58:30.219274 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.219210 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.219975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.219950 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:58:30.282623 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.220639 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.289719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.289688 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:58:30.387600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.225811 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.387600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.316738 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:30.427442 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.425381 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:30.427442 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.425448 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.427442 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.426316 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.559016 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.558918 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:30.568734 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.568684 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.60:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.616833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.616773 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:58:30.686385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.686342 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.702596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.698870 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.730603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.723497 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:58:30.730603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.723563 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:58:30.844773 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.844668 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:30.884637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:30.861806 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 17:58:31.276652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.231799 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:31.408601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.305368 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:58:31.441599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.366259 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:31.467599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.366344 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:58:31.592314 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.436706 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:31.912964 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.912854 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:31.935601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.923643 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.418728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:31.984697 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.419555 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.135907 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.471604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.457698 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:32.471604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.457805 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.480787 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.480727 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.534973 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.534929 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:58:32.537516 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.537484 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.538403 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.538373 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.592730 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.592683 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:32.759600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:32.752006 4980 patch_prober.go:29] interesting pod/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:33.214607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.201471 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.214607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.168130 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.243600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.158462 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:58:33.282610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.255647 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:33.282610 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.255714 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.521719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.429908 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.554661 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:58:33.435200 4980 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.132s" Nov 03 17:58:33.564182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.502924 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": context deadline exceeded" Nov 03 17:58:33.611601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.538694 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.624121 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.541799 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.625127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.544619 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:33.636395 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.542038 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.699808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.593358 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.817657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.816705 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:33.817657 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.816771 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.856748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.856699 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:33.925428 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:33.925393 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:58:34.454086 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.432607 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": context deadline exceeded" Nov 03 17:58:34.454086 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.433413 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:58:34.454086 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.433749 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 17:58:34.487024 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.486985 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 03 17:58:34.640625 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.632100 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:58:34.675664 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.671505 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:34.694903 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.694862 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 17:58:34.702928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.702883 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:34.733928 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.704318 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:58:34.841447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.714634 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:34.842471 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.714687 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:58:34.850663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:34.716251 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:58:35.335603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.335364 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 17:58:35.426607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.385004 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:58:35.426607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.407256 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:58:35.426607 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.420787 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:58:35.444778 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.443648 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:35.469608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.468442 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:58:35.540135 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.540104 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:58:35.603440 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.603358 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:58:35.626316 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.626277 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:58:35.630015 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:35.629994 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:58:36.972447 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:36.970860 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 17:58:36.990633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:36.990599 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 17:58:37.147930 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.147895 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 17:58:37.157959 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.157934 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.169754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.169725 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 17:58:37.213124 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.213096 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-aa500779-openshift-default-dockercfg-57tdw\"" Nov 03 17:58:37.248737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.248642 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.248737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.248711 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7drm2\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.248927 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.248850 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249002 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249232 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249210 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249429 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249410 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249629 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249545 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249750 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249645 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.249801 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.249764 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351667 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351627 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351831 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351720 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351831 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351817 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351843 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351875 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.351975 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351916 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352130 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352049 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352214 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352184 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352275 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352240 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352328 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.351944 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352328 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352274 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352455 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352438 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352508 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352495 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7drm2\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.352561 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.352523 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.354380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.354356 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.354946 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.354928 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.422596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.416330 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7drm2\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.422596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.422263 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\") pod \"e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:37.479094 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:37.477874 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:38.172803 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:38.172772 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 17:58:38.173415 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:38.173393 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod891a371e_7bec_49a8_9300_0d3bc0c9ec57.slice/crio-8aa33d2080e656927b4c0af8b58996669ddbc924df0c594b65ba70fa311313d0 WatchSource:0}: Error finding container 8aa33d2080e656927b4c0af8b58996669ddbc924df0c594b65ba70fa311313d0: Status 404 returned error can't find the container with id 8aa33d2080e656927b4c0af8b58996669ddbc924df0c594b65ba70fa311313d0 Nov 03 17:58:38.176539 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:38.176497 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:38.176669 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:38.176633 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:38.176720 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:38.176682 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:39.016328 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:39.016295 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerStarted","Data":"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab"} Nov 03 17:58:39.016504 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:39.016397 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerStarted","Data":"8aa33d2080e656927b4c0af8b58996669ddbc924df0c594b65ba70fa311313d0"} Nov 03 17:58:39.192137 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:39.192085 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podStartSLOduration=2.192069902 podStartE2EDuration="2.192069902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:39.189171459 +0000 UTC m=+1499.018255063" watchObservedRunningTime="2026-11-03 17:58:39.192069902 +0000 UTC m=+1499.021153505" Nov 03 17:58:39.478633 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:39.478566 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:39.484086 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:39.484063 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:40.023097 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.023071 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:40.024159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.024082 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 17:58:40.504689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.504660 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:58:40.505385 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.505361 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 17:58:40.509902 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.509883 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:58:40.510343 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:40.510327 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 17:58:42.057800 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.057766 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 17:58:42.063828 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.063812 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.107850 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.107824 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 17:58:42.112149 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.112130 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.160993 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.160968 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161036 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161111 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161232 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161187 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161505 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161481 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161728 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.161868 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161816 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.162235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.161984 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.162235 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.162043 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xvjj\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.166348 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.166319 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 17:58:42.221856 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.221823 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 17:58:42.263524 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263490 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.263754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263543 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.263754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263640 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.263754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263737 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.263950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263789 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.263950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263810 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.263950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263860 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.263950 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.263878 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9gvp\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.264245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264043 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264046 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264126 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264157 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264245 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264223 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264259 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.264527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264305 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xvjj\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264238 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264337 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.264527 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264512 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.264815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264621 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.264815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264620 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264708 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.264815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.264796 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.265134 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.265063 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.266437 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.266415 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.268407 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.268388 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.328233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.328169 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.328233 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.328193 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xvjj\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.366418 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366381 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366418 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366419 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366465 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366480 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9gvp\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366583 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366630 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366693 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366658 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.366966 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366706 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.366998 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367081 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.367015 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367134 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.367070 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367192 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.367136 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.367563 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.367904 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.367880 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.369022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.368996 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.369022 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.369013 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.378140 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.378122 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:42.430338 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.430296 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.430499 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.430444 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9gvp\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\") pod \"e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.725438 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.725411 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:42.823698 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.823631 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 17:58:42.825053 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:42.825026 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e6d0a17_3921_48ef_8120_8b66793229a7.slice/crio-5864f42cbaa1c927325b87ad14ce0d77b487540ec2082920899facaf19e568df WatchSource:0}: Error finding container 5864f42cbaa1c927325b87ad14ce0d77b487540ec2082920899facaf19e568df: Status 404 returned error can't find the container with id 5864f42cbaa1c927325b87ad14ce0d77b487540ec2082920899facaf19e568df Nov 03 17:58:42.828405 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.828363 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:42.828519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.828455 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:42.828519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:42.828500 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:43.054270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.054186 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" event={"ID":"0e6d0a17-3921-48ef-8120-8b66793229a7","Type":"ContainerStarted","Data":"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54"} Nov 03 17:58:43.054270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.054257 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" event={"ID":"0e6d0a17-3921-48ef-8120-8b66793229a7","Type":"ContainerStarted","Data":"5864f42cbaa1c927325b87ad14ce0d77b487540ec2082920899facaf19e568df"} Nov 03 17:58:43.115362 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:43.115333 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb873742f_df2f_4e9d_8432_be1a7b4b3bbf.slice/crio-c1e5aa17c89f8dbc3dfe57a9f2168c5144ad9454fc694270aca749f291262a99 WatchSource:0}: Error finding container c1e5aa17c89f8dbc3dfe57a9f2168c5144ad9454fc694270aca749f291262a99: Status 404 returned error can't find the container with id c1e5aa17c89f8dbc3dfe57a9f2168c5144ad9454fc694270aca749f291262a99 Nov 03 17:58:43.116226 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.116200 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 17:58:43.118386 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.118355 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:43.118488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.118428 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:43.118488 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.118462 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:43.182127 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.182084 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podStartSLOduration=1.182068948 podStartE2EDuration="1.182068948s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:43.178937448 +0000 UTC m=+1503.008021051" watchObservedRunningTime="2026-11-03 17:58:43.182068948 +0000 UTC m=+1503.011152554" Nov 03 17:58:43.381808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.381262 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:43.381808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.381396 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:43.381808 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:43.381448 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:44.113612 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.113581 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" event={"ID":"b873742f-df2f-4e9d-8432-be1a7b4b3bbf","Type":"ContainerStarted","Data":"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3"} Nov 03 17:58:44.114537 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.114518 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" event={"ID":"b873742f-df2f-4e9d-8432-be1a7b4b3bbf","Type":"ContainerStarted","Data":"c1e5aa17c89f8dbc3dfe57a9f2168c5144ad9454fc694270aca749f291262a99"} Nov 03 17:58:44.243118 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.243064 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podStartSLOduration=2.2430508160000002 podStartE2EDuration="2.243050816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:44.24049106 +0000 UTC m=+1504.069574664" watchObservedRunningTime="2026-11-03 17:58:44.243050816 +0000 UTC m=+1504.072134483" Nov 03 17:58:44.382660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.382042 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:44.382660 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.382110 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:44.537656 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:58:44.537615 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:58:44.726401 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.726365 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:44.726891 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.726870 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:44.726949 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:44.726930 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:45.384597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:45.380943 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:45.384597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:45.380999 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:45.725848 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:45.725809 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:45.726148 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:45.725882 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:46.382073 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:46.382029 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:46.382379 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:46.382348 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:46.726695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:46.726659 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:46.726695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:46.726722 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:47.381596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:47.380957 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:47.381596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:47.381022 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:47.726731 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:47.726700 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:47.726900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:47.726754 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:48.448598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.446256 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:48.448598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.446336 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:48.568597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.542758 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 17:58:48.670604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.632793 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 17:58:48.671056 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.671027 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:48.749087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.712925 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 17:58:48.749087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.717338 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6jks\" (UniqueName: \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:48.749087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.717475 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:48.749087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.742068 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-aa500779-dockercfg-ts8w6\"" Nov 03 17:58:48.749087 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.742165 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-aa500779\"" Nov 03 17:58:49.003678 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:48.999456 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:49.059731 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.059679 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:49.060373 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.015827 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6jks\" (UniqueName: \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:49.070815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.070775 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:49.110597 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.079942 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:49.166136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.145804 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 17:58:49.166415 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.145886 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.190600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.162423 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6jks\" (UniqueName: \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\") pod \"payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:49.254849 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.254756 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:49.323923 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.305364 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2l85v\" (UniqueName: \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.324782 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.324743 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.406499 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.406448 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:49.406726 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.406541 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:49.412785 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.412763 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:49.426143 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.426105 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2l85v\" (UniqueName: \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.426433 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.426412 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.427501 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.427477 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.510648 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.509853 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2l85v\" (UniqueName: \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\") pod \"payload-processing-e2e-derive-aa500779-85bddccb7-szsmq\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.605287 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.604126 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:49.726822 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.726453 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:49.726822 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.726536 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:49.887030 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:49.886644 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 17:58:50.065791 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.065755 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 17:58:50.066369 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:50.066341 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a12a2dd_f3fe_47d7_8638_efac8854f203.slice/crio-4157d4e75e94d1ed91c653085ef3f5b335ac00aa7d0bd334d5af599a2864eac2 WatchSource:0}: Error finding container 4157d4e75e94d1ed91c653085ef3f5b335ac00aa7d0bd334d5af599a2864eac2: Status 404 returned error can't find the container with id 4157d4e75e94d1ed91c653085ef3f5b335ac00aa7d0bd334d5af599a2864eac2 Nov 03 17:58:50.353658 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.353612 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" event={"ID":"0a12a2dd-f3fe-47d7-8638-efac8854f203","Type":"ContainerStarted","Data":"94eeeb1be9738669fa6e99ea06dd8af29f7e1b6f72507addc8a20d7386e28c5d"} Nov 03 17:58:50.353857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.353690 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:58:50.353857 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.353718 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" event={"ID":"0a12a2dd-f3fe-47d7-8638-efac8854f203","Type":"ContainerStarted","Data":"4157d4e75e94d1ed91c653085ef3f5b335ac00aa7d0bd334d5af599a2864eac2"} Nov 03 17:58:50.355663 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.355625 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" event={"ID":"be871039-67ab-439c-9dd5-c6bbd01b369e","Type":"ContainerStarted","Data":"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616"} Nov 03 17:58:50.355799 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.355703 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:58:50.355799 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.355729 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" event={"ID":"be871039-67ab-439c-9dd5-c6bbd01b369e","Type":"ContainerStarted","Data":"2424bbef09100a800c61720611544104a8b6f042c70273d98aa87044c8525a68"} Nov 03 17:58:50.378763 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.378735 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" start-of-body= Nov 03 17:58:50.378870 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.378793 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.90:15021/healthz/ready\": dial tcp 10.132.0.90:15021: connect: connection refused" Nov 03 17:58:50.483727 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.483665 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" podStartSLOduration=2.4836460750000002 podStartE2EDuration="2.483646075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:50.476828696 +0000 UTC m=+1510.305912294" watchObservedRunningTime="2026-11-03 17:58:50.483646075 +0000 UTC m=+1510.312729683" Nov 03 17:58:50.628647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.628295 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" podStartSLOduration=2.62827404 podStartE2EDuration="2.62827404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:50.625008846 +0000 UTC m=+1510.454092450" watchObservedRunningTime="2026-11-03 17:58:50.62827404 +0000 UTC m=+1510.457357645" Nov 03 17:58:50.731091 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.731050 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body= Nov 03 17:58:50.731387 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:50.731353 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" Nov 03 17:58:51.605289 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.605253 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:51.606097 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.606073 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:51.607117 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.607094 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 17:58:51.731398 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.731373 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:51.731708 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.731688 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:51.732462 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:51.732446 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 17:58:52.359456 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.359421 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 17:58:52.365228 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.365198 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.424197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.424157 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-c74b2c26-serving-cert\"" Nov 03 17:58:52.429934 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.429899 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 17:58:52.500261 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.500229 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.500647 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.500612 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.500900 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.500878 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjhx6\" (UniqueName: \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.500990 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.500955 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.602374 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.602333 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.602555 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.602501 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.602709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.602664 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjhx6\" (UniqueName: \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.602781 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.602741 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.602913 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:58:52.602889 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-serving-cert: secret "maas-api-e2e-migrate-c74b2c26-serving-cert" not found Nov 03 17:58:52.602990 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:58:52.602966 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls podName:d86835a0-7747-4057-9482-807952f10a30 nodeName:}" failed. No retries permitted until 2026-11-03 17:58:53.102941914 +0000 UTC m=+1512.932025509 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls") pod "maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" (UID: "d86835a0-7747-4057-9482-807952f10a30") : secret "maas-api-e2e-migrate-c74b2c26-serving-cert" not found Nov 03 17:58:52.603334 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.603309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.605697 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.605666 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:52.669108 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:52.669034 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjhx6\" (UniqueName: \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:53.125595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:53.125103 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:53.137542 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:53.137508 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") pod \"maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:53.291598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:53.287249 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:54.917603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:54.905288 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.054728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.043908 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.154617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.142133 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.313626 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.306057 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 17:58:55.313626 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.306628 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 17:58:55.567537 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.567439 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" event={"ID":"d86835a0-7747-4057-9482-807952f10a30","Type":"ContainerStarted","Data":"12c3b0d5ff305b117957f62df919f159985b0f2708f9f0637e036ea8d85e739f"} Nov 03 17:58:55.653728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.653680 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.676721 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.676666 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.834649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.808840 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:55.853599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.840778 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:55.877659 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.877621 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 17:58:55.973623 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:55.924815 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:56.083076 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.083031 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:58:56.120007 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.119920 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:58:56.120792 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.120772 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.243599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.240841 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:56.243599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.241729 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:58:56.304380 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.300650 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:58:56.421617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.413489 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.453023 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.452982 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.453361 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.453338 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.453475 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.453460 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.453713 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.453693 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.457639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.453975 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.534200 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.534137 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.534586 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.534532 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbwh7\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.534864 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.534826 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.557603 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.427376 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:56.708652 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.679416 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:56.750595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.708645 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:56.761624 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.708692 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.710492 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbwh7\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.766992 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767145 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767195 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767305 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767330 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767418 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767533 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.767601 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.768138 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.768853 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.769617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.769107 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.811719 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.811684 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.814068 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.812725 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:56.815841 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.815798 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 17:58:56.852283 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.852003 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:58:56.876210 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.876184 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:58:56.941617 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.937828 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:58:57.302606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.296801 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:58:57.302606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.296858 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:58:57.302606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:56.937660 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:58:57.318606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.317179 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:58:57.341602 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.068750 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbwh7\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:57.341602 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.098964 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": context deadline exceeded" Nov 03 17:58:57.341602 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.115917 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:57.355649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.068217 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:57.355649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.015164 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:57.355649 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.036134 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\") pod \"e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:57.432936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.399833 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:58:57.432936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.400856 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:58:57.432936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.402497 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:58:57.432936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.406336 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:58:57.432936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.407389 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:58:57.437948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.437918 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:58:57.480882 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.478662 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:58:57.507839 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.507174 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:58:57.581600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.577671 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:58:57.581600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.581235 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:58:57.622797 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.622770 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:57.889861 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.889747 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:58:57.966634 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:57.961780 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:58:58.220728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.190713 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:58:58.220728 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.215453 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:58:58.242872 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.242842 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" event={"ID":"d86835a0-7747-4057-9482-807952f10a30","Type":"ContainerStarted","Data":"fad3017385c921229e41c53e4519218eb9deccf7217defff15c50024bab34d47"} Nov 03 17:58:58.243167 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.243149 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:58:58.638320 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:58:58.638270 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod776e5da1_9243_4086_8032_1951428d3b67.slice/crio-c28cd0b65bd9b0b2e06d8befcf3a06f711f623ccd46b06670628559341c99397 WatchSource:0}: Error finding container c28cd0b65bd9b0b2e06d8befcf3a06f711f623ccd46b06670628559341c99397: Status 404 returned error can't find the container with id c28cd0b65bd9b0b2e06d8befcf3a06f711f623ccd46b06670628559341c99397 Nov 03 17:58:58.643269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.643240 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 17:58:58.643431 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.643404 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:58.643495 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.643463 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:58.643557 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:58.643493 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:58:59.257935 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.257891 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" event={"ID":"776e5da1-9243-4086-8032-1951428d3b67","Type":"ContainerStarted","Data":"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18"} Nov 03 17:58:59.258136 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.257966 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" event={"ID":"776e5da1-9243-4086-8032-1951428d3b67","Type":"ContainerStarted","Data":"c28cd0b65bd9b0b2e06d8befcf3a06f711f623ccd46b06670628559341c99397"} Nov 03 17:58:59.401361 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.401298 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" podStartSLOduration=5.480861978 podStartE2EDuration="7.40127658s" totalImagesPullingTime="1.920414602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:59.395926834 +0000 UTC m=+1519.225010447" watchObservedRunningTime="2026-11-03 17:58:59.40127658 +0000 UTC m=+1519.230360184" Nov 03 17:58:59.631709 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.627429 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:58:59.637639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.627561 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:58:59.637639 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:58:59.636682 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:00.624821 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:00.624785 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:00.625201 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:00.624841 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:01.389755 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:01.389726 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 17:59:01.390269 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:01.390253 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 17:59:01.509489 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:01.509414 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podStartSLOduration=6.509390161 podStartE2EDuration="6.509390161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:58:59.536284034 +0000 UTC m=+1519.365367638" watchObservedRunningTime="2026-11-03 17:59:01.509390161 +0000 UTC m=+1521.338473806" Nov 03 17:59:01.632242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:01.632140 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:01.632242 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:01.632214 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:02.135598 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.130784 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 17:59:02.186668 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.180909 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.261604 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.255478 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 17:59:02.300637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.280689 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.336710 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.336669 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.347595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.342064 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.364771 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.364918 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs2f6\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.364960 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.365087 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.365176 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.368599 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.365202 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.606596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.598225 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-12a446f9-openshift-default-dockercfg-tj2vx\"" Nov 03 17:59:02.632897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.632412 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:02.632897 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.632480 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.634521 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.634696 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs2f6\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.634734 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.634994 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.635125 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.635153 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.635284 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.635309 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.635387 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.641941 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.657998 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.658957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.659244 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.676125 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.659525 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.730080 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.727200 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs2f6\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.851595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.814870 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.853062 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.852483 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:02.889091 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:02.888976 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\") pod \"e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:03.029085 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.029054 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:03.515396 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.515368 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 17:59:03.516278 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:59:03.516247 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6a6372_3795_41df_b1b0_b8a91b6f9c0f.slice/crio-973fb8d4ffc0368fb5979cb11c42b89ec78ea5a0561225b4a33090b5d7effa7f WatchSource:0}: Error finding container 973fb8d4ffc0368fb5979cb11c42b89ec78ea5a0561225b4a33090b5d7effa7f: Status 404 returned error can't find the container with id 973fb8d4ffc0368fb5979cb11c42b89ec78ea5a0561225b4a33090b5d7effa7f Nov 03 17:59:03.519275 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.519236 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:03.519383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.519318 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:03.519383 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.519353 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:03.625689 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.625659 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:03.625815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.625721 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:03.751695 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.751647 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerStarted","Data":"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb"} Nov 03 17:59:03.752204 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.751768 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerStarted","Data":"973fb8d4ffc0368fb5979cb11c42b89ec78ea5a0561225b4a33090b5d7effa7f"} Nov 03 17:59:03.892282 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:03.889593 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podStartSLOduration=1.889549745 podStartE2EDuration="1.889549745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:03.886949332 +0000 UTC m=+1523.716032954" watchObservedRunningTime="2026-11-03 17:59:03.889549745 +0000 UTC m=+1523.718633355" Nov 03 17:59:04.065608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:04.057543 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:04.065608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:04.058903 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 17:59:04.065608 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:04.058961 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 17:59:04.643819 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:04.642188 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:04.643819 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:04.642261 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:05.036472 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.036430 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:05.298843 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.298681 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 17:59:05.625446 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.625354 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" start-of-body= Nov 03 17:59:05.625446 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.625422 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.95:15021/healthz/ready\": dial tcp 10.132.0.95:15021: connect: connection refused" Nov 03 17:59:05.848757 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.848713 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:05.855011 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:05.854982 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 17:59:07.216716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:07.216676 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:59:07.217530 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:07.217489 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:59:07.218321 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:07.218300 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 17:59:10.051353 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.051321 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 17:59:10.057072 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.057052 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069093 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069070 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069207 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069115 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069270 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069243 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069316 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069276 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069444 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069424 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069515 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069502 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069589 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069530 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069683 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069663 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6phz\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.069764 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.069748 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.112699 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.112673 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\"" Nov 03 17:59:10.166815 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.166779 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 17:59:10.171537 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171508 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171560 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171662 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171754 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171693 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171816 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171907 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171932 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.171994 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.171966 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6phz\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.172246 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.172077 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.172748 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.172723 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.174676 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.172970 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.174846 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.173113 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.174948 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.173941 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.175070 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.174194 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.175262 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.175236 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.176182 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.176159 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.237614 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.237562 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.238216 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.238190 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6phz\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\") pod \"maas-default-gateway-openshift-default-5df48fcc99-sg8bl\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.377070 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.376936 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:10.883020 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.882979 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 17:59:10.883132 ip-10-0-139-234 kubenswrapper[4980]: W1103 17:59:10.883052 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66cc9f76_7e43_45ce_9998_7a6fbf224073.slice/crio-fe1c483082e1e1dcce86b147324586aaeb55b6fcefb0c5c62273a15d2c936868 WatchSource:0}: Error finding container fe1c483082e1e1dcce86b147324586aaeb55b6fcefb0c5c62273a15d2c936868: Status 404 returned error can't find the container with id fe1c483082e1e1dcce86b147324586aaeb55b6fcefb0c5c62273a15d2c936868 Nov 03 17:59:10.885772 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.885743 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:10.885833 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.885809 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:10.885875 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.885837 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:10.940238 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:10.940207 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" event={"ID":"66cc9f76-7e43-45ce-9998-7a6fbf224073","Type":"ContainerStarted","Data":"fe1c483082e1e1dcce86b147324586aaeb55b6fcefb0c5c62273a15d2c936868"} Nov 03 17:59:11.949071 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:11.949040 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" event={"ID":"66cc9f76-7e43-45ce-9998-7a6fbf224073","Type":"ContainerStarted","Data":"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5"} Nov 03 17:59:12.125002 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:12.124943 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podStartSLOduration=2.124928133 podStartE2EDuration="2.124928133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 17:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 17:59:12.123190157 +0000 UTC m=+1531.952273760" watchObservedRunningTime="2026-11-03 17:59:12.124928133 +0000 UTC m=+1531.954011734" Nov 03 17:59:12.378659 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:12.377946 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:12.388725 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:12.388684 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:12.388865 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:12.388736 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:13.379622 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:13.377957 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:13.379622 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:13.378033 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:14.379601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:14.377943 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:14.379601 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:14.378010 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:14.525423 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:59:14.525392 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:59:15.379737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:15.379694 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:15.379956 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:15.379764 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:16.537600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:16.537447 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:16.537600 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:16.537555 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:17.377838 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:17.377787 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:17.378110 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:17.377864 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:18.378178 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:18.378141 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" start-of-body= Nov 03 17:59:18.378637 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:18.378205 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": dial tcp 10.132.0.97:15021: connect: connection refused" Nov 03 17:59:20.187341 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:20.187296 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:20.188197 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:20.188172 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:20.188892 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:20.188869 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 17:59:39.408499 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:39.408450 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:39.410532 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:39.410502 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:59:39.430716 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:39.430670 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:39.441543 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:39.440671 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:39.521856 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:39.521811 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.080458 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.079158 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:59:40.258403 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.257358 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.414840 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.414736 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.437737 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.437702 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:59:40.454735 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.447259 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.456111 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.456088 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:59:40.538653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.534672 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.538653 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.535266 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:59:40.971596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.968993 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:40.971596 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:40.969388 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:59:41.079983 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.079939 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:59:41.081283 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.081258 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:59:41.313770 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.313657 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:59:41.411508 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.385911 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:59:41.411508 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.409023 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:59:41.437177 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.413251 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:59:41.459714 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.459671 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:41.508682 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.508638 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:41.539352 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:41.539307 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:42.223606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.223087 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 17:59:42.223606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.223511 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 17:59:42.241606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.240225 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:59:42.241606 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.240471 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:59:42.326736 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.310490 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 17:59:42.383159 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.383114 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 17:59:42.383483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.383456 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 17:59:42.384249 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.384225 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:59:42.407268 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.407224 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:42.588321 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.587190 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:42.588321 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.587600 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 17:59:42.588917 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.588890 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:42.591595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:42.590283 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 17:59:43.273674 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.273627 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.296174 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.290479 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.329918 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.329875 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.355118 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.355077 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.407816 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.407753 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 17:59:43.492258 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.450857 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.541220 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:59:43.468339 4980 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.054s" Nov 03 17:59:43.541519 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.541501 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.599758 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.600355 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.600415 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.600451 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.600484 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 17:59:43.603595 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.600515 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 17:59:43.629590 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.618026 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 17:59:43.629590 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.625502 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 17:59:43.636071 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.636043 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 17:59:43.680609 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:43.680086 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 17:59:44.625442 ip-10-0-139-234 kubenswrapper[4980]: E1103 17:59:44.625406 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 17:59:51.785556 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:51.785234 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": read tcp 10.132.0.2:47766->10.132.0.82:15021: read: connection reset by peer" start-of-body= Nov 03 17:59:51.785556 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:51.785352 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": read tcp 10.132.0.2:47766->10.132.0.82:15021: read: connection reset by peer" Nov 03 17:59:52.611217 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.611184 4980 generic.go:356] "Generic (PLEG): container finished" podID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerID="e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba" exitCode=0 Nov 03 17:59:52.611444 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.611243 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerDied","Data":"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba"} Nov 03 17:59:52.611767 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.611741 4980 scope.go:122] "RemoveContainer" containerID="e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba" Nov 03 17:59:52.613483 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.613451 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:52.613565 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.613524 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:52.613565 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:52.613557 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 17:59:53.620658 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:53.620624 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerStarted","Data":"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f"} Nov 03 17:59:53.724896 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:53.724863 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:59:53.725122 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:53.725102 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:53.725173 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:53.725150 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:54.725551 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:54.725513 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:54.726004 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:54.725607 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:55.725588 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:55.725541 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:55.726017 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:55.725639 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:56.724893 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:56.724860 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:56.725071 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:56.724925 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:57.724936 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:57.724897 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:57.725347 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:57.724964 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:58.725259 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:58.725224 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 17:59:58.725685 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:58.725273 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 17:59:59.729855 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:59.729820 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:59:59.730355 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:59.730285 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 17:59:59.731196 ip-10-0-139-234 kubenswrapper[4980]: I1103 17:59:59.731173 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 18:00:00.194551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.194522 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:00.197821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.197804 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.208561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.208537 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:00.212501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.212483 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.220682 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.220659 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:00.223710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.223690 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.244517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.244488 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.244709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.244684 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzccd\" (UniqueName: \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.244781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.244735 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgkjl\" (UniqueName: \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.244892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.244875 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.244953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.244923 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcz4j\" (UniqueName: \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.245145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.245122 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.282951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.282911 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:00.305965 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.305933 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:00.306120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.306021 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:00.331289 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.331198 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:00:00.347400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347362 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.347644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347451 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcz4j\" (UniqueName: \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.347644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347629 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.347790 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347764 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.348870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347893 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzccd\" (UniqueName: \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.348870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.347953 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgkjl\" (UniqueName: \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.348870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.348142 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.348870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.348593 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.349165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.349105 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.411903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.411871 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcz4j\" (UniqueName: \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\") pod \"maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.412052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.412010 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgkjl\" (UniqueName: \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\") pod \"maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.412094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.412010 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzccd\" (UniqueName: \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\") pod \"maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.512935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.512855 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:00.526698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.526669 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:00.538224 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.538194 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:00.915542 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:00:00.915511 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod314ac5bc_555d_4c4b_b047_fa8fcfad21d0.slice/crio-afcad5a0ac29ab166de9cd622bcebcce57cefa88d241c193b9ce4af287b66247 WatchSource:0}: Error finding container afcad5a0ac29ab166de9cd622bcebcce57cefa88d241c193b9ce4af287b66247: Status 404 returned error can't find the container with id afcad5a0ac29ab166de9cd622bcebcce57cefa88d241c193b9ce4af287b66247 Nov 03 18:00:00.915898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.915805 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:00.969808 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:00:00.969211 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21a48369_b354_4337_a78d_de96c4fac30e.slice/crio-947540685f1465a615627d960850acb31585004da20300d2a90421675a668ca5 WatchSource:0}: Error finding container 947540685f1465a615627d960850acb31585004da20300d2a90421675a668ca5: Status 404 returned error can't find the container with id 947540685f1465a615627d960850acb31585004da20300d2a90421675a668ca5 Nov 03 18:00:00.969808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.969539 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:00.974889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:00.974866 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:00.975030 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:00:00.975004 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70944a3e_c6ae_4056_b9bb_2f44a423eead.slice/crio-722384e938d7a15ba5c94230f684dfbee3d863352669d6835550de2cd5786922 WatchSource:0}: Error finding container 722384e938d7a15ba5c94230f684dfbee3d863352669d6835550de2cd5786922: Status 404 returned error can't find the container with id 722384e938d7a15ba5c94230f684dfbee3d863352669d6835550de2cd5786922 Nov 03 18:00:01.745524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.745424 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerStarted","Data":"1647712bad0fb2d0955d09a819840fb8beaec2f1e52090304390b8c35c91cf84"} Nov 03 18:00:01.745524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.745518 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerStarted","Data":"afcad5a0ac29ab166de9cd622bcebcce57cefa88d241c193b9ce4af287b66247"} Nov 03 18:00:01.747289 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.747257 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerStarted","Data":"b665c8f0f491713b11e0f8d7785dabc715a2ef08e5715d7790a2fe7ea701062c"} Nov 03 18:00:01.747421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.747309 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerStarted","Data":"722384e938d7a15ba5c94230f684dfbee3d863352669d6835550de2cd5786922"} Nov 03 18:00:01.749189 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.749164 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerStarted","Data":"d5f63b0f94ed45ef23ba5e0f1287d8a01e9a9a742257ef362cf2a6ff19308139"} Nov 03 18:00:01.749311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.749201 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerStarted","Data":"947540685f1465a615627d960850acb31585004da20300d2a90421675a668ca5"} Nov 03 18:00:01.881644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:01.881521 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" podStartSLOduration=1.88149984 podStartE2EDuration="1.88149984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:01.877928676 +0000 UTC m=+1581.707012281" watchObservedRunningTime="2026-11-03 18:00:01.88149984 +0000 UTC m=+1581.710583443" Nov 03 18:00:02.004772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.004658 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" podStartSLOduration=2.004643336 podStartE2EDuration="2.004643336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.00154292 +0000 UTC m=+1581.830626522" watchObservedRunningTime="2026-11-03 18:00:02.004643336 +0000 UTC m=+1581.833726938" Nov 03 18:00:02.122623 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.122541 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" podStartSLOduration=2.122528604 podStartE2EDuration="2.122528604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:00:02.121173873 +0000 UTC m=+1581.950257476" watchObservedRunningTime="2026-11-03 18:00:02.122528604 +0000 UTC m=+1581.951612206" Nov 03 18:00:02.759311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.759278 4980 generic.go:356] "Generic (PLEG): container finished" podID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerID="1647712bad0fb2d0955d09a819840fb8beaec2f1e52090304390b8c35c91cf84" exitCode=60 Nov 03 18:00:02.759520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.759320 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerDied","Data":"1647712bad0fb2d0955d09a819840fb8beaec2f1e52090304390b8c35c91cf84"} Nov 03 18:00:02.759778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.759756 4980 scope.go:122] "RemoveContainer" containerID="1647712bad0fb2d0955d09a819840fb8beaec2f1e52090304390b8c35c91cf84" Nov 03 18:00:02.761162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.761138 4980 generic.go:356] "Generic (PLEG): container finished" podID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerID="b665c8f0f491713b11e0f8d7785dabc715a2ef08e5715d7790a2fe7ea701062c" exitCode=60 Nov 03 18:00:02.761250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.761218 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerDied","Data":"b665c8f0f491713b11e0f8d7785dabc715a2ef08e5715d7790a2fe7ea701062c"} Nov 03 18:00:02.761452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.761443 4980 scope.go:122] "RemoveContainer" containerID="b665c8f0f491713b11e0f8d7785dabc715a2ef08e5715d7790a2fe7ea701062c" Nov 03 18:00:02.763093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.763066 4980 generic.go:356] "Generic (PLEG): container finished" podID="21a48369-b354-4337-a78d-de96c4fac30e" containerID="d5f63b0f94ed45ef23ba5e0f1287d8a01e9a9a742257ef362cf2a6ff19308139" exitCode=60 Nov 03 18:00:02.763188 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.763123 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerDied","Data":"d5f63b0f94ed45ef23ba5e0f1287d8a01e9a9a742257ef362cf2a6ff19308139"} Nov 03 18:00:02.763414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:02.763400 4980 scope.go:122] "RemoveContainer" containerID="d5f63b0f94ed45ef23ba5e0f1287d8a01e9a9a742257ef362cf2a6ff19308139" Nov 03 18:00:03.774404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.774364 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerStarted","Data":"50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924"} Nov 03 18:00:03.774856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.774814 4980 scope.go:122] "RemoveContainer" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" Nov 03 18:00:03.775269 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:03.775247 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr_odh-ai-gateway-infra(314ac5bc-555d-4c4b-b047-fa8fcfad21d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" Nov 03 18:00:03.776633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.776608 4980 generic.go:356] "Generic (PLEG): container finished" podID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerID="f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4" exitCode=60 Nov 03 18:00:03.776739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.776693 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerDied","Data":"f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4"} Nov 03 18:00:03.776790 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.776755 4980 scope.go:122] "RemoveContainer" containerID="b665c8f0f491713b11e0f8d7785dabc715a2ef08e5715d7790a2fe7ea701062c" Nov 03 18:00:03.777113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.777076 4980 scope.go:122] "RemoveContainer" containerID="f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4" Nov 03 18:00:03.777540 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:03.777516 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9_odh-ai-gateway-infra(70944a3e-c6ae-4056-b9bb-2f44a423eead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" Nov 03 18:00:03.779047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:03.779021 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerStarted","Data":"c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b"} Nov 03 18:00:04.790072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.790033 4980 generic.go:356] "Generic (PLEG): container finished" podID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" exitCode=60 Nov 03 18:00:04.790615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.790119 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerDied","Data":"50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924"} Nov 03 18:00:04.790615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.790177 4980 scope.go:122] "RemoveContainer" containerID="1647712bad0fb2d0955d09a819840fb8beaec2f1e52090304390b8c35c91cf84" Nov 03 18:00:04.790615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.790482 4980 scope.go:122] "RemoveContainer" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" Nov 03 18:00:04.791016 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:04.790987 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr_odh-ai-gateway-infra(314ac5bc-555d-4c4b-b047-fa8fcfad21d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" Nov 03 18:00:04.792404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.792386 4980 scope.go:122] "RemoveContainer" containerID="f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4" Nov 03 18:00:04.792796 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:04.792764 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9_odh-ai-gateway-infra(70944a3e-c6ae-4056-b9bb-2f44a423eead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" Nov 03 18:00:04.793996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.793974 4980 generic.go:356] "Generic (PLEG): container finished" podID="21a48369-b354-4337-a78d-de96c4fac30e" containerID="c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b" exitCode=60 Nov 03 18:00:04.794106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.794032 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerDied","Data":"c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b"} Nov 03 18:00:04.794295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.794279 4980 scope.go:122] "RemoveContainer" containerID="c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b" Nov 03 18:00:04.794668 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:04.794647 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp_odh-ai-gateway-infra(21a48369-b354-4337-a78d-de96c4fac30e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" podUID="21a48369-b354-4337-a78d-de96c4fac30e" Nov 03 18:00:04.803550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:04.803528 4980 scope.go:122] "RemoveContainer" containerID="d5f63b0f94ed45ef23ba5e0f1287d8a01e9a9a742257ef362cf2a6ff19308139" Nov 03 18:00:05.802936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:05.802901 4980 scope.go:122] "RemoveContainer" containerID="c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b" Nov 03 18:00:05.803451 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:05.803305 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp_odh-ai-gateway-infra(21a48369-b354-4337-a78d-de96c4fac30e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" podUID="21a48369-b354-4337-a78d-de96c4fac30e" Nov 03 18:00:05.804727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:05.804709 4980 scope.go:122] "RemoveContainer" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" Nov 03 18:00:05.805029 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:05.805010 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr_odh-ai-gateway-infra(314ac5bc-555d-4c4b-b047-fa8fcfad21d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" Nov 03 18:00:14.290425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:14.288328 4980 scope.go:122] "RemoveContainer" containerID="c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b" Nov 03 18:00:14.582354 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:14.582272 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:00:14.935278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:14.935192 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerStarted","Data":"d6eb3dcc3a1c47bdc5dba675df9535b1daae1f26c67cd41edbf3134ce447f6f3"} Nov 03 18:00:14.966916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:14.966884 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 18:00:15.074823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:15.073301 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 18:00:15.944194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:15.944162 4980 generic.go:356] "Generic (PLEG): container finished" podID="21a48369-b354-4337-a78d-de96c4fac30e" containerID="d6eb3dcc3a1c47bdc5dba675df9535b1daae1f26c67cd41edbf3134ce447f6f3" exitCode=60 Nov 03 18:00:15.944712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:15.944201 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerDied","Data":"d6eb3dcc3a1c47bdc5dba675df9535b1daae1f26c67cd41edbf3134ce447f6f3"} Nov 03 18:00:15.944712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:15.944260 4980 scope.go:122] "RemoveContainer" containerID="c4ba20be449b95b76d83d83c13a7ae03bdbb69ad5fd6fd0307a7b7f104bc4b6b" Nov 03 18:00:15.944808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:15.944717 4980 scope.go:122] "RemoveContainer" containerID="d6eb3dcc3a1c47bdc5dba675df9535b1daae1f26c67cd41edbf3134ce447f6f3" Nov 03 18:00:15.945057 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:15.945035 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp_odh-ai-gateway-infra(21a48369-b354-4337-a78d-de96c4fac30e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" podUID="21a48369-b354-4337-a78d-de96c4fac30e" Nov 03 18:00:16.092863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:16.092829 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:16.287344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:16.287242 4980 scope.go:122] "RemoveContainer" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" Nov 03 18:00:16.287664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:16.287639 4980 scope.go:122] "RemoveContainer" containerID="f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4" Nov 03 18:00:17.163723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.163684 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerStarted","Data":"73647ed7709b625d0000af5b18cf10f2f361e80b1e04758ec087d142935c27f3"} Nov 03 18:00:17.442809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.442783 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:17.484664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.484635 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\") pod \"21a48369-b354-4337-a78d-de96c4fac30e\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " Nov 03 18:00:17.484812 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.484803 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\" (UniqueName: \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\") pod \"21a48369-b354-4337-a78d-de96c4fac30e\" (UID: \"21a48369-b354-4337-a78d-de96c4fac30e\") " Nov 03 18:00:17.484990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.484969 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca" pod "21a48369-b354-4337-a78d-de96c4fac30e" (UID: "21a48369-b354-4337-a78d-de96c4fac30e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:17.485459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.485437 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/21a48369-b354-4337-a78d-de96c4fac30e-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:17.487842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.487819 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j" pod "21a48369-b354-4337-a78d-de96c4fac30e" (UID: "21a48369-b354-4337-a78d-de96c4fac30e"). InnerVolumeSpecName "kube-api-access-zcz4j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:17.587953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:17.587915 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zcz4j\" (UniqueName: \"kubernetes.io/projected/21a48369-b354-4337-a78d-de96c4fac30e-kube-api-access-zcz4j\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:18.228213 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.228178 4980 generic.go:356] "Generic (PLEG): container finished" podID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerID="73647ed7709b625d0000af5b18cf10f2f361e80b1e04758ec087d142935c27f3" exitCode=60 Nov 03 18:00:18.228707 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.228259 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerDied","Data":"73647ed7709b625d0000af5b18cf10f2f361e80b1e04758ec087d142935c27f3"} Nov 03 18:00:18.228707 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.228343 4980 scope.go:122] "RemoveContainer" containerID="50dac5488ffeae2d94615cc1b2b0fb9688d56b270dfb45178b3c1f4b3efa7924" Nov 03 18:00:18.228885 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.228868 4980 scope.go:122] "RemoveContainer" containerID="73647ed7709b625d0000af5b18cf10f2f361e80b1e04758ec087d142935c27f3" Nov 03 18:00:18.229320 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:18.229292 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr_odh-ai-gateway-infra(314ac5bc-555d-4c4b-b047-fa8fcfad21d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" Nov 03 18:00:18.230753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.230733 4980 generic.go:356] "Generic (PLEG): container finished" podID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerID="4d3a573ebbc17c15ed64862b81470dc33c5031acc9a23ee77f0e9a91a73b8bce" exitCode=60 Nov 03 18:00:18.230851 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.230806 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerDied","Data":"4d3a573ebbc17c15ed64862b81470dc33c5031acc9a23ee77f0e9a91a73b8bce"} Nov 03 18:00:18.231258 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.231234 4980 scope.go:122] "RemoveContainer" containerID="4d3a573ebbc17c15ed64862b81470dc33c5031acc9a23ee77f0e9a91a73b8bce" Nov 03 18:00:18.231667 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:18.231633 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9_odh-ai-gateway-infra(70944a3e-c6ae-4056-b9bb-2f44a423eead)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" Nov 03 18:00:18.232456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.232438 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" Nov 03 18:00:18.232456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.232448 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp" event={"ID":"21a48369-b354-4337-a78d-de96c4fac30e","Type":"ContainerDied","Data":"947540685f1465a615627d960850acb31585004da20300d2a90421675a668ca5"} Nov 03 18:00:18.243213 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.243192 4980 scope.go:122] "RemoveContainer" containerID="f9099729c94d7480193094cdbefb679dc126769c7416eda757074ed290d7bed4" Nov 03 18:00:18.256408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.256390 4980 scope.go:122] "RemoveContainer" containerID="d6eb3dcc3a1c47bdc5dba675df9535b1daae1f26c67cd41edbf3134ce447f6f3" Nov 03 18:00:18.428416 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.428384 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:18.653220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.653127 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:18.663409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:18.663379 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-c74b2c26-29895480-shnlp"] Nov 03 18:00:19.465549 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.465523 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:19.517824 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.517741 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\") pod \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " Nov 03 18:00:19.517981 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.517926 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\" (UniqueName: \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\") pod \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\" (UID: \"314ac5bc-555d-4c4b-b047-fa8fcfad21d0\") " Nov 03 18:00:19.518253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.518204 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca" pod "314ac5bc-555d-4c4b-b047-fa8fcfad21d0" (UID: "314ac5bc-555d-4c4b-b047-fa8fcfad21d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:19.518786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.518762 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:19.520413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.520389 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd" pod "314ac5bc-555d-4c4b-b047-fa8fcfad21d0" (UID: "314ac5bc-555d-4c4b-b047-fa8fcfad21d0"). InnerVolumeSpecName "kube-api-access-vzccd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:19.554133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.554099 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:19.621377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.621343 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzccd\" (UniqueName: \"kubernetes.io/projected/314ac5bc-555d-4c4b-b047-fa8fcfad21d0-kube-api-access-vzccd\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:19.767201 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.767175 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:19.825552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.825465 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\") pod \"70944a3e-c6ae-4056-b9bb-2f44a423eead\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " Nov 03 18:00:19.825552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.825505 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\" (UniqueName: \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\") pod \"70944a3e-c6ae-4056-b9bb-2f44a423eead\" (UID: \"70944a3e-c6ae-4056-b9bb-2f44a423eead\") " Nov 03 18:00:19.825946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.825924 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca" pod "70944a3e-c6ae-4056-b9bb-2f44a423eead" (UID: "70944a3e-c6ae-4056-b9bb-2f44a423eead"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:19.826260 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.826239 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/70944a3e-c6ae-4056-b9bb-2f44a423eead-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:19.828043 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.828022 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl" pod "70944a3e-c6ae-4056-b9bb-2f44a423eead" (UID: "70944a3e-c6ae-4056-b9bb-2f44a423eead"). InnerVolumeSpecName "kube-api-access-zgkjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:19.928153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.928111 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgkjl\" (UniqueName: \"kubernetes.io/projected/70944a3e-c6ae-4056-b9bb-2f44a423eead-kube-api-access-zgkjl\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:19.968094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:19.968034 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" podUID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" containerName="payload-processing" containerID="cri-o://ff845c7d12f843da871b0b7d9167ea5f7318eff92f789857e9406b596d894c7e" gracePeriod=25 Nov 03 18:00:20.074207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.074155 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" podUID="6a381410-0dab-449b-bcfc-b45b2b924329" containerName="payload-pre-processing" containerID="cri-o://4c73e5aa0cc25f94346ab59764a8b716fb38fbb26e316c1131290760d1719da3" gracePeriod=25 Nov 03 18:00:20.262615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.262532 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" event={"ID":"314ac5bc-555d-4c4b-b047-fa8fcfad21d0","Type":"ContainerDied","Data":"afcad5a0ac29ab166de9cd622bcebcce57cefa88d241c193b9ce4af287b66247"} Nov 03 18:00:20.262781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.262660 4980 scope.go:122] "RemoveContainer" containerID="73647ed7709b625d0000af5b18cf10f2f361e80b1e04758ec087d142935c27f3" Nov 03 18:00:20.262844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.262798 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr" Nov 03 18:00:20.267468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.267432 4980 generic.go:356] "Generic (PLEG): container finished" podID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" containerID="ff845c7d12f843da871b0b7d9167ea5f7318eff92f789857e9406b596d894c7e" exitCode=0 Nov 03 18:00:20.267637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.267539 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" event={"ID":"6ed52d39-d4ce-4301-a087-a3e2e43e68a1","Type":"ContainerDied","Data":"ff845c7d12f843da871b0b7d9167ea5f7318eff92f789857e9406b596d894c7e"} Nov 03 18:00:20.269756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.269727 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" event={"ID":"70944a3e-c6ae-4056-b9bb-2f44a423eead","Type":"ContainerDied","Data":"722384e938d7a15ba5c94230f684dfbee3d863352669d6835550de2cd5786922"} Nov 03 18:00:20.270045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.270024 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9" Nov 03 18:00:20.272518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.272491 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 18:00:20.278825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.278793 4980 generic.go:356] "Generic (PLEG): container finished" podID="6a381410-0dab-449b-bcfc-b45b2b924329" containerID="4c73e5aa0cc25f94346ab59764a8b716fb38fbb26e316c1131290760d1719da3" exitCode=0 Nov 03 18:00:20.278909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.278841 4980 scope.go:122] "RemoveContainer" containerID="4d3a573ebbc17c15ed64862b81470dc33c5031acc9a23ee77f0e9a91a73b8bce" Nov 03 18:00:20.278909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.278891 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" event={"ID":"6a381410-0dab-449b-bcfc-b45b2b924329","Type":"ContainerDied","Data":"4c73e5aa0cc25f94346ab59764a8b716fb38fbb26e316c1131290760d1719da3"} Nov 03 18:00:20.301254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.301225 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21a48369-b354-4337-a78d-de96c4fac30e" path="/var/lib/kubelet/pods/21a48369-b354-4337-a78d-de96c4fac30e/volumes" Nov 03 18:00:20.340190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.340084 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\" (UniqueName: \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\") pod \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " Nov 03 18:00:20.340427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.340413 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\") pod \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\" (UID: \"6ed52d39-d4ce-4301-a087-a3e2e43e68a1\") " Nov 03 18:00:20.341668 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.341635 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume" pod "6ed52d39-d4ce-4301-a087-a3e2e43e68a1" (UID: "6ed52d39-d4ce-4301-a087-a3e2e43e68a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:20.351252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.351219 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf" pod "6ed52d39-d4ce-4301-a087-a3e2e43e68a1" (UID: "6ed52d39-d4ce-4301-a087-a3e2e43e68a1"). InnerVolumeSpecName "kube-api-access-wp2nf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:20.412678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.412654 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 18:00:20.443034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.443001 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:20.443034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.443034 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wp2nf\" (UniqueName: \"kubernetes.io/projected/6ed52d39-d4ce-4301-a087-a3e2e43e68a1-kube-api-access-wp2nf\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:20.544122 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.544090 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\") pod \"6a381410-0dab-449b-bcfc-b45b2b924329\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " Nov 03 18:00:20.544509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.544247 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\" (UniqueName: \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\") pod \"6a381410-0dab-449b-bcfc-b45b2b924329\" (UID: \"6a381410-0dab-449b-bcfc-b45b2b924329\") " Nov 03 18:00:20.544509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.544450 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume" pod "6a381410-0dab-449b-bcfc-b45b2b924329" (UID: "6a381410-0dab-449b-bcfc-b45b2b924329"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:20.544764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.544747 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a381410-0dab-449b-bcfc-b45b2b924329-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:20.546632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.546609 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq" pod "6a381410-0dab-449b-bcfc-b45b2b924329" (UID: "6a381410-0dab-449b-bcfc-b45b2b924329"). InnerVolumeSpecName "kube-api-access-8zdvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:20.560231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.560206 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:20.569266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.569238 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-776ba7-29895480-vnqlr"] Nov 03 18:00:20.646523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.646428 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zdvq\" (UniqueName: \"kubernetes.io/projected/6a381410-0dab-449b-bcfc-b45b2b924329-kube-api-access-8zdvq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:20.683781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.683749 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:20.689980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:20.689951 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-aa500779-29895480-sn7d9"] Nov 03 18:00:21.290337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.290303 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" event={"ID":"6a381410-0dab-449b-bcfc-b45b2b924329","Type":"ContainerDied","Data":"000a38472eff3c014a3835b5e49ab5a014637da36b1d2515d409376912a567e3"} Nov 03 18:00:21.290526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.290367 4980 scope.go:122] "RemoveContainer" containerID="4c73e5aa0cc25f94346ab59764a8b716fb38fbb26e316c1131290760d1719da3" Nov 03 18:00:21.290526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.290306 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5" Nov 03 18:00:21.293817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.293789 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" event={"ID":"6ed52d39-d4ce-4301-a087-a3e2e43e68a1","Type":"ContainerDied","Data":"e2e83141d022d7f7d1a7043bb7b6457e24b906b5e5de1ab62c61321ca2044666"} Nov 03 18:00:21.293945 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.293893 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg" Nov 03 18:00:21.305871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.305843 4980 scope.go:122] "RemoveContainer" containerID="ff845c7d12f843da871b0b7d9167ea5f7318eff92f789857e9406b596d894c7e" Nov 03 18:00:21.462253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.462219 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 18:00:21.512495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.512458 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-c74b2c26-cf44668ff-j2gd5"] Nov 03 18:00:21.581755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.581663 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 18:00:21.589060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:21.589023 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-c74b2c26-d67d4b7-f88gg"] Nov 03 18:00:22.295332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:22.295301 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" path="/var/lib/kubelet/pods/314ac5bc-555d-4c4b-b047-fa8fcfad21d0/volumes" Nov 03 18:00:22.295801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:22.295786 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a381410-0dab-449b-bcfc-b45b2b924329" path="/var/lib/kubelet/pods/6a381410-0dab-449b-bcfc-b45b2b924329/volumes" Nov 03 18:00:22.296369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:22.296350 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" path="/var/lib/kubelet/pods/6ed52d39-d4ce-4301-a087-a3e2e43e68a1/volumes" Nov 03 18:00:22.297917 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:22.297889 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" path="/var/lib/kubelet/pods/70944a3e-c6ae-4056-b9bb-2f44a423eead/volumes" Nov 03 18:00:27.501306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:27.501273 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 18:00:27.633599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:27.610143 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 18:00:32.503593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:32.502637 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" podUID="8fb202ae-af49-4b2d-b612-c1fce977126d" containerName="payload-processing" containerID="cri-o://fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5" gracePeriod=25 Nov 03 18:00:32.611653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:32.611133 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" podUID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" containerName="payload-pre-processing" containerID="cri-o://0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065" gracePeriod=25 Nov 03 18:00:33.024447 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.024411 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 18:00:33.150328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.149351 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\") pod \"8fb202ae-af49-4b2d-b612-c1fce977126d\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " Nov 03 18:00:33.150328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.149505 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\" (UniqueName: \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\") pod \"8fb202ae-af49-4b2d-b612-c1fce977126d\" (UID: \"8fb202ae-af49-4b2d-b612-c1fce977126d\") " Nov 03 18:00:33.177598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.176161 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume" pod "8fb202ae-af49-4b2d-b612-c1fce977126d" (UID: "8fb202ae-af49-4b2d-b612-c1fce977126d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:33.276676 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.271325 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fb202ae-af49-4b2d-b612-c1fce977126d-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:33.296212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.295717 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg" pod "8fb202ae-af49-4b2d-b612-c1fce977126d" (UID: "8fb202ae-af49-4b2d-b612-c1fce977126d"). InnerVolumeSpecName "kube-api-access-vsbsg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:33.397891 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.397691 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsbsg\" (UniqueName: \"kubernetes.io/projected/8fb202ae-af49-4b2d-b612-c1fce977126d-kube-api-access-vsbsg\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:33.444619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.442213 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 18:00:33.466656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.457610 4980 generic.go:356] "Generic (PLEG): container finished" podID="8fb202ae-af49-4b2d-b612-c1fce977126d" containerID="fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5" exitCode=0 Nov 03 18:00:33.466656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.457825 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" event={"ID":"8fb202ae-af49-4b2d-b612-c1fce977126d","Type":"ContainerDied","Data":"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5"} Nov 03 18:00:33.466656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.457881 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" event={"ID":"8fb202ae-af49-4b2d-b612-c1fce977126d","Type":"ContainerDied","Data":"f786db901d3aea97ebd2b02bbbfe22380fe484d385553b17fc98477c5418d341"} Nov 03 18:00:33.466656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.457911 4980 scope.go:122] "RemoveContainer" containerID="fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5" Nov 03 18:00:33.466656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.458081 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9" Nov 03 18:00:33.523820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.523485 4980 generic.go:356] "Generic (PLEG): container finished" podID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" containerID="0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065" exitCode=0 Nov 03 18:00:33.523820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.523552 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" event={"ID":"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a","Type":"ContainerDied","Data":"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065"} Nov 03 18:00:33.523820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.523669 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" event={"ID":"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a","Type":"ContainerDied","Data":"295bcaab4624b8edf0743eea2069a1466ec846f6fc931f8cbf8ce9c940b20000"} Nov 03 18:00:33.523820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.523758 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2" Nov 03 18:00:33.547687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.547665 4980 scope.go:122] "RemoveContainer" containerID="fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5" Nov 03 18:00:33.548040 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:33.548019 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5\": container with ID starting with fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5 not found: ID does not exist" containerID="fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5" Nov 03 18:00:33.548130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.548056 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5"} err="failed to get container status \"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5\": rpc error: code = NotFound desc = could not find container \"fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5\": container with ID starting with fbec6d861ec994389f554003f8335bbf54ead1985bd72f12a246d1876ab742d5 not found: ID does not exist" Nov 03 18:00:33.548130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.548085 4980 scope.go:122] "RemoveContainer" containerID="0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065" Nov 03 18:00:33.604598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.602190 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\") pod \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " Nov 03 18:00:33.604598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.602346 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\" (UniqueName: \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\") pod \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\" (UID: \"a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a\") " Nov 03 18:00:33.615657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.609816 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume" pod "a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" (UID: "a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:00:33.627384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.627201 4980 scope.go:122] "RemoveContainer" containerID="0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065" Nov 03 18:00:33.627673 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:33.627643 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065\": container with ID starting with 0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065 not found: ID does not exist" containerID="0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065" Nov 03 18:00:33.627791 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.627689 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065"} err="failed to get container status \"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065\": rpc error: code = NotFound desc = could not find container \"0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065\": container with ID starting with 0431284073d0315ec5a760711ba686f6d0122a2f28a07b3056cda66d8e65b065 not found: ID does not exist" Nov 03 18:00:33.634329 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.634290 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492" pod "a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" (UID: "a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a"). InnerVolumeSpecName "kube-api-access-j5492". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:00:33.712166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.712127 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:33.712422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.712400 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5492\" (UniqueName: \"kubernetes.io/projected/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a-kube-api-access-j5492\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:00:33.715531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.715491 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 18:00:33.723334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:33.723239 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-1c1ea78c-585bb8b94f-7zcz9"] Nov 03 18:00:34.009845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:34.009383 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 18:00:34.014273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:34.014235 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-1c1ea78c-5dfc8ccd67-v9pd2"] Nov 03 18:00:34.315474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:34.315396 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb202ae-af49-4b2d-b612-c1fce977126d" path="/var/lib/kubelet/pods/8fb202ae-af49-4b2d-b612-c1fce977126d/volumes" Nov 03 18:00:34.315988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:34.315970 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" path="/var/lib/kubelet/pods/a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a/volumes" Nov 03 18:00:41.065810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:41.065762 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:00:41.066237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:41.065844 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:00:42.958589 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:42.958503 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rk2t6" podUID="3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2" containerName="postgres" probeResult="failure" output="command timed out" Nov 03 18:00:44.574246 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:00:44.574204 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:00:47.784870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.784834 4980 generic.go:356] "Generic (PLEG): container finished" podID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerID="966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab" exitCode=0 Nov 03 18:00:47.786298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.784895 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerDied","Data":"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab"} Nov 03 18:00:47.786298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.785429 4980 scope.go:122] "RemoveContainer" containerID="966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab" Nov 03 18:00:47.787233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.787202 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:47.787334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.787293 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:47.787397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:47.787345 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:48.801634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.801553 4980 generic.go:356] "Generic (PLEG): container finished" podID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerID="936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb" exitCode=0 Nov 03 18:00:48.802076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.801712 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerDied","Data":"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb"} Nov 03 18:00:48.802183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.802166 4980 scope.go:122] "RemoveContainer" containerID="936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb" Nov 03 18:00:48.806118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.805546 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:48.806118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.805689 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:48.806118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.805733 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:00:48.806530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:48.806507 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerStarted","Data":"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3"} Nov 03 18:00:49.480442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:49.479997 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:00:49.480442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:49.480125 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:49.480442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:49.480181 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:49.815900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:49.815793 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerStarted","Data":"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807"} Nov 03 18:00:50.030050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.030012 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:00:50.030784 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.030759 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:50.030894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.030798 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:50.487597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.482494 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:50.487597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.482562 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:50.849392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:50.849314 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:00:51.029892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:51.029856 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:51.030067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:51.029911 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:51.479596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:51.478941 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:51.479596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:51.479009 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:52.029421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:52.029385 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:52.029839 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:52.029438 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:52.482064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:52.482012 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:52.482350 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:52.482321 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:53.029434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:53.029401 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:53.029892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:53.029455 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:53.487822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:53.487781 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:53.488093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:53.488066 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:54.029714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:54.029663 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:54.030222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:54.029762 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:54.480005 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:54.479968 4980 patch_prober.go:29] interesting pod/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body= Nov 03 18:00:54.480268 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:54.480235 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" Nov 03 18:00:55.028802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:55.023694 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:00:55.053701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:55.053670 4980 patch_prober.go:29] interesting pod/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" start-of-body= Nov 03 18:00:55.054286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:55.054252 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.96:15021/healthz/ready\": dial tcp 10.132.0.96:15021: connect: connection refused" Nov 03 18:00:55.483855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:55.483827 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:00:55.862055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:55.861977 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:00:56.034694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:56.034663 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:00:56.035786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:56.035766 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:00:59.392465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:59.392423 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 18:00:59.393030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:59.392664 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" podUID="d86835a0-7747-4057-9482-807952f10a30" containerName="maas-api" containerID="cri-o://fad3017385c921229e41c53e4519218eb9deccf7217defff15c50024bab34d47" gracePeriod=30 Nov 03 18:00:59.997656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:59.997047 4980 generic.go:356] "Generic (PLEG): container finished" podID="d86835a0-7747-4057-9482-807952f10a30" containerID="fad3017385c921229e41c53e4519218eb9deccf7217defff15c50024bab34d47" exitCode=0 Nov 03 18:00:59.998612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:59.997878 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" event={"ID":"d86835a0-7747-4057-9482-807952f10a30","Type":"ContainerDied","Data":"fad3017385c921229e41c53e4519218eb9deccf7217defff15c50024bab34d47"} Nov 03 18:00:59.998612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:00:59.997978 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 18:01:00.070212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.070170 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 18:01:00.117148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.117124 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 18:01:00.190345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.186293 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") pod \"d86835a0-7747-4057-9482-807952f10a30\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " Nov 03 18:01:00.190729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.190695 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\") pod \"d86835a0-7747-4057-9482-807952f10a30\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " Nov 03 18:01:00.190879 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.190753 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\") pod \"d86835a0-7747-4057-9482-807952f10a30\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " Nov 03 18:01:00.190879 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.190860 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\" (UniqueName: \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\") pod \"d86835a0-7747-4057-9482-807952f10a30\" (UID: \"d86835a0-7747-4057-9482-807952f10a30\") " Nov 03 18:01:00.191155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.191119 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca" pod "d86835a0-7747-4057-9482-807952f10a30" (UID: "d86835a0-7747-4057-9482-807952f10a30"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:00.191961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.191939 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d86835a0-7747-4057-9482-807952f10a30-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:00.194100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.194079 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls" pod "d86835a0-7747-4057-9482-807952f10a30" (UID: "d86835a0-7747-4057-9482-807952f10a30"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:00.194660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.194635 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs" pod "d86835a0-7747-4057-9482-807952f10a30" (UID: "d86835a0-7747-4057-9482-807952f10a30"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:01:00.194660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.194643 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6" pod "d86835a0-7747-4057-9482-807952f10a30" (UID: "d86835a0-7747-4057-9482-807952f10a30"). InnerVolumeSpecName "kube-api-access-cjhx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:00.293456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.293377 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:00.293456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.293405 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjhx6\" (UniqueName: \"kubernetes.io/projected/d86835a0-7747-4057-9482-807952f10a30-kube-api-access-cjhx6\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:00.293456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:00.293415 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d86835a0-7747-4057-9482-807952f10a30-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:01.005669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:01.005632 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" Nov 03 18:01:01.005669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:01.005649 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw" event={"ID":"d86835a0-7747-4057-9482-807952f10a30","Type":"ContainerDied","Data":"12c3b0d5ff305b117957f62df919f159985b0f2708f9f0637e036ea8d85e739f"} Nov 03 18:01:01.006230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:01.005722 4980 scope.go:122] "RemoveContainer" containerID="fad3017385c921229e41c53e4519218eb9deccf7217defff15c50024bab34d47" Nov 03 18:01:01.178714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:01.178681 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 18:01:01.183515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:01.183483 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-c74b2c26-c449bbd45-ltfjw"] Nov 03 18:01:02.323939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:02.323899 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d86835a0-7747-4057-9482-807952f10a30" path="/var/lib/kubelet/pods/d86835a0-7747-4057-9482-807952f10a30/volumes" Nov 03 18:01:03.265855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.265813 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:01:03.273242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273209 4980 cpu_manager.go:414] "containerMap: removing container" podUID="d86835a0-7747-4057-9482-807952f10a30" containerName="maas-api" Nov 03 18:01:03.273467 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273450 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d86835a0-7747-4057-9482-807952f10a30" containerName="maas-api" Nov 03 18:01:03.273608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273595 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6a381410-0dab-449b-bcfc-b45b2b924329" containerName="payload-pre-processing" Nov 03 18:01:03.273722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273708 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a381410-0dab-449b-bcfc-b45b2b924329" containerName="payload-pre-processing" Nov 03 18:01:03.273802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273793 4980 cpu_manager.go:414] "containerMap: removing container" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.273873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273863 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.273951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.273941 4980 cpu_manager.go:414] "containerMap: removing container" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.274030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.274019 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.295678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.295636 4980 cpu_manager.go:414] "containerMap: removing container" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.295950 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.295930 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.296092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296080 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" containerName="payload-processing" Nov 03 18:01:03.296202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296190 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" containerName="payload-processing" Nov 03 18:01:03.296334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296322 4980 cpu_manager.go:414] "containerMap: removing container" podUID="8fb202ae-af49-4b2d-b612-c1fce977126d" containerName="payload-processing" Nov 03 18:01:03.296433 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296422 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb202ae-af49-4b2d-b612-c1fce977126d" containerName="payload-processing" Nov 03 18:01:03.296525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296516 4980 cpu_manager.go:414] "containerMap: removing container" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.296634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296623 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.296743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296733 4980 cpu_manager.go:414] "containerMap: removing container" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.296834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296823 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.296936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.296926 4980 cpu_manager.go:414] "containerMap: removing container" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.297032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.297020 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.297130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.297120 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" containerName="payload-pre-processing" Nov 03 18:01:03.297232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.297220 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" containerName="payload-pre-processing" Nov 03 18:01:03.552669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.529773 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.553749 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.553721 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ed52d39-d4ce-4301-a087-a3e2e43e68a1" containerName="payload-processing" Nov 03 18:01:03.553981 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.553960 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb202ae-af49-4b2d-b612-c1fce977126d" containerName="payload-processing" Nov 03 18:01:03.554097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.554085 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.563611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.554180 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:03.563858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.563835 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.563977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.563964 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:03.564083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.564072 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2fc39aa-5d93-4cb4-aac2-90b0c9fc601a" containerName="payload-pre-processing" Nov 03 18:01:03.564183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.564172 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:03.586691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.586659 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d86835a0-7747-4057-9482-807952f10a30" containerName="maas-api" Nov 03 18:01:03.595634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.587139 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a381410-0dab-449b-bcfc-b45b2b924329" containerName="payload-pre-processing" Nov 03 18:01:03.662256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.662223 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.668702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.668669 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:01:03.753992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.750694 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-3e138c04-openshift-default-dockercfg-hkdjn\"" Nov 03 18:01:03.899899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.899818 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.899899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.899857 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900070 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900109 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900132 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900310 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900402 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900376 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7xpm\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900418 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:03.900513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:03.900501 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.001825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.001784 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.001825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.001823 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.001959 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002004 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002031 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002102 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002163 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7xpm\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002256 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002353 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002513 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002733 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002791 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.002873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.002832 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.004709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.004684 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.005158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.005140 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.065868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.065829 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7xpm\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.066339 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.066320 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\") pod \"e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.098270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.098245 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:04.541844 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:01:04.541813 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d47cb0_cb62_4ae4_aa0a_348421ca18d6.slice/crio-dee54f7b3e57ee41120ca93161c1978da1b172377378e145b6fe5c9419497197 WatchSource:0}: Error finding container dee54f7b3e57ee41120ca93161c1978da1b172377378e145b6fe5c9419497197: Status 404 returned error can't find the container with id dee54f7b3e57ee41120ca93161c1978da1b172377378e145b6fe5c9419497197 Nov 03 18:01:04.542020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.541996 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:01:04.544788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.544756 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:04.544867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.544819 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:04.544867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.544849 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:04.998906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:04.998865 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" podUID="0a12a2dd-f3fe-47d7-8638-efac8854f203" containerName="payload-processing" containerID="cri-o://94eeeb1be9738669fa6e99ea06dd8af29f7e1b6f72507addc8a20d7386e28c5d" gracePeriod=25 Nov 03 18:01:05.071346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.071296 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" podUID="be871039-67ab-439c-9dd5-c6bbd01b369e" containerName="payload-pre-processing" containerID="cri-o://2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616" gracePeriod=25 Nov 03 18:01:05.158861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.158823 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerStarted","Data":"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c"} Nov 03 18:01:05.159026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.158876 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerStarted","Data":"dee54f7b3e57ee41120ca93161c1978da1b172377378e145b6fe5c9419497197"} Nov 03 18:01:05.160492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.160469 4980 generic.go:356] "Generic (PLEG): container finished" podID="0a12a2dd-f3fe-47d7-8638-efac8854f203" containerID="94eeeb1be9738669fa6e99ea06dd8af29f7e1b6f72507addc8a20d7386e28c5d" exitCode=0 Nov 03 18:01:05.160634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.160542 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" event={"ID":"0a12a2dd-f3fe-47d7-8638-efac8854f203","Type":"ContainerDied","Data":"94eeeb1be9738669fa6e99ea06dd8af29f7e1b6f72507addc8a20d7386e28c5d"} Nov 03 18:01:05.377504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.377480 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 18:01:05.387922 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.387886 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 18:01:05.424630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.424600 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\" (UniqueName: \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\") pod \"be871039-67ab-439c-9dd5-c6bbd01b369e\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " Nov 03 18:01:05.424794 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.424658 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\") pod \"be871039-67ab-439c-9dd5-c6bbd01b369e\" (UID: \"be871039-67ab-439c-9dd5-c6bbd01b369e\") " Nov 03 18:01:05.424866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.424849 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\" (UniqueName: \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\") pod \"0a12a2dd-f3fe-47d7-8638-efac8854f203\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " Nov 03 18:01:05.424933 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.424883 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\") pod \"0a12a2dd-f3fe-47d7-8638-efac8854f203\" (UID: \"0a12a2dd-f3fe-47d7-8638-efac8854f203\") " Nov 03 18:01:05.425049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.425022 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume" pod "be871039-67ab-439c-9dd5-c6bbd01b369e" (UID: "be871039-67ab-439c-9dd5-c6bbd01b369e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:05.425276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.425259 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume" pod "0a12a2dd-f3fe-47d7-8638-efac8854f203" (UID: "0a12a2dd-f3fe-47d7-8638-efac8854f203"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:05.425438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.425420 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0a12a2dd-f3fe-47d7-8638-efac8854f203-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:05.425493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.425446 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be871039-67ab-439c-9dd5-c6bbd01b369e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:05.426835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.426815 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks" pod "be871039-67ab-439c-9dd5-c6bbd01b369e" (UID: "be871039-67ab-439c-9dd5-c6bbd01b369e"). InnerVolumeSpecName "kube-api-access-n6jks". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:05.427139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.427123 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v" pod "0a12a2dd-f3fe-47d7-8638-efac8854f203" (UID: "0a12a2dd-f3fe-47d7-8638-efac8854f203"). InnerVolumeSpecName "kube-api-access-2l85v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:05.534422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.527208 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2l85v\" (UniqueName: \"kubernetes.io/projected/0a12a2dd-f3fe-47d7-8638-efac8854f203-kube-api-access-2l85v\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:05.534422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.527248 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n6jks\" (UniqueName: \"kubernetes.io/projected/be871039-67ab-439c-9dd5-c6bbd01b369e-kube-api-access-n6jks\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:05.546706 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:05.541716 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podStartSLOduration=2.541692773 podStartE2EDuration="2.541692773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:05.382791981 +0000 UTC m=+1645.211875594" watchObservedRunningTime="2026-11-03 18:01:05.541692773 +0000 UTC m=+1645.370776376" Nov 03 18:01:06.098734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.098706 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:06.104255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.104227 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:06.170210 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.170177 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" Nov 03 18:01:06.170210 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.170193 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq" event={"ID":"0a12a2dd-f3fe-47d7-8638-efac8854f203","Type":"ContainerDied","Data":"4157d4e75e94d1ed91c653085ef3f5b335ac00aa7d0bd334d5af599a2864eac2"} Nov 03 18:01:06.170491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.170256 4980 scope.go:122] "RemoveContainer" containerID="94eeeb1be9738669fa6e99ea06dd8af29f7e1b6f72507addc8a20d7386e28c5d" Nov 03 18:01:06.177731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.177703 4980 generic.go:356] "Generic (PLEG): container finished" podID="be871039-67ab-439c-9dd5-c6bbd01b369e" containerID="2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616" exitCode=0 Nov 03 18:01:06.177967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.177759 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" Nov 03 18:01:06.178094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.177775 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" event={"ID":"be871039-67ab-439c-9dd5-c6bbd01b369e","Type":"ContainerDied","Data":"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616"} Nov 03 18:01:06.178146 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.178089 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx" event={"ID":"be871039-67ab-439c-9dd5-c6bbd01b369e","Type":"ContainerDied","Data":"2424bbef09100a800c61720611544104a8b6f042c70273d98aa87044c8525a68"} Nov 03 18:01:06.179051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.178597 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:06.179670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.179648 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:01:06.189853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.189684 4980 scope.go:122] "RemoveContainer" containerID="2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616" Nov 03 18:01:06.200322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.200307 4980 scope.go:122] "RemoveContainer" containerID="2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616" Nov 03 18:01:06.200615 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:06.200599 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616\": container with ID starting with 2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616 not found: ID does not exist" containerID="2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616" Nov 03 18:01:06.200665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.200625 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616"} err="failed to get container status \"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616\": rpc error: code = NotFound desc = could not find container \"2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616\": container with ID starting with 2fdfeb1c23108d4f337d76da77aca47c701e82eed5d05bdc064d007b0a039616 not found: ID does not exist" Nov 03 18:01:06.343781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.343709 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 18:01:06.354901 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.354877 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-aa500779-85bddccb7-szsmq"] Nov 03 18:01:06.597019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.596916 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 18:01:06.604084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:06.604042 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-aa500779-c49df7b59-mw8vx"] Nov 03 18:01:08.305627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:08.305594 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a12a2dd-f3fe-47d7-8638-efac8854f203" path="/var/lib/kubelet/pods/0a12a2dd-f3fe-47d7-8638-efac8854f203/volumes" Nov 03 18:01:08.306109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:08.306094 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be871039-67ab-439c-9dd5-c6bbd01b369e" path="/var/lib/kubelet/pods/be871039-67ab-439c-9dd5-c6bbd01b369e/volumes" Nov 03 18:01:09.443783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:09.443748 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 18:01:09.444196 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:09.443979 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" containerID="cri-o://feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f" gracePeriod=30 Nov 03 18:01:10.668400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.668360 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:01:10.669368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669350 4980 cpu_manager.go:414] "containerMap: removing container" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:10.669430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669371 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:10.669430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669386 4980 cpu_manager.go:414] "containerMap: removing container" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:10.669430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669394 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669446 4980 cpu_manager.go:414] "containerMap: removing container" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669454 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669466 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0a12a2dd-f3fe-47d7-8638-efac8854f203" containerName="payload-processing" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669473 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a12a2dd-f3fe-47d7-8638-efac8854f203" containerName="payload-processing" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669511 4980 cpu_manager.go:414] "containerMap: removing container" podUID="be871039-67ab-439c-9dd5-c6bbd01b369e" containerName="payload-pre-processing" Nov 03 18:01:10.669539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669519 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be871039-67ab-439c-9dd5-c6bbd01b369e" containerName="payload-pre-processing" Nov 03 18:01:10.669743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669686 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="314ac5bc-555d-4c4b-b047-fa8fcfad21d0" containerName="cleanup" Nov 03 18:01:10.669743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669700 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a12a2dd-f3fe-47d7-8638-efac8854f203" containerName="payload-processing" Nov 03 18:01:10.669743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669709 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="21a48369-b354-4337-a78d-de96c4fac30e" containerName="cleanup" Nov 03 18:01:10.669743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669723 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="70944a3e-c6ae-4056-b9bb-2f44a423eead" containerName="cleanup" Nov 03 18:01:10.669743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.669735 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="be871039-67ab-439c-9dd5-c6bbd01b369e" containerName="payload-pre-processing" Nov 03 18:01:10.677823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.677541 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.742156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.742116 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-394741e0-openshift-default-dockercfg-hxf6x\"" Nov 03 18:01:10.787887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.787844 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.903076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.903046 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.903425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.903402 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9tv7\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.903713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.903689 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.904193 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.904175 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.904402 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.904387 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.928456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.928342 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.928823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.928798 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.929106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.929086 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:10.963840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:10.963801 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031197 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031286 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031321 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9tv7\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031414 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031445 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031485 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031517 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031538 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.031632 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.033219 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.033495 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.036595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.033755 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.040590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.037948 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.040590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.038232 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.062595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.043112 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.062595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.053704 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.101345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.101313 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9tv7\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.106669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.106588 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:11.373459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:11.373191 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:12.009805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.006495 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:01:12.106758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.106644 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:01:12.118902 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.116018 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:12.118902 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.116111 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:12.118902 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.116157 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:01:12.250723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.228185 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:01:12.250723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.228312 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.284463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.284435 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3e138c04-dockercfg-pkmns\"" Nov 03 18:01:12.284885 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.284858 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3e138c04\"" Nov 03 18:01:12.423088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.423041 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.423312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.423291 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h9tm\" (UniqueName: \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.429948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.429892 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:01:12.441808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.441752 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerStarted","Data":"157c703a7f6d24b8bc4c1d710087892e93af1ff6959affc09bcc048ba7e37a9d"} Nov 03 18:01:12.442441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.442402 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:01:12.442792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.442323 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.527068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.526132 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h9tm\" (UniqueName: \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.527068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.526241 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.527068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.526300 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vll86\" (UniqueName: \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.527068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.526441 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.529166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.529139 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.591894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.591805 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h9tm\" (UniqueName: \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\") pod \"payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.628895 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.628858 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.629075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.628914 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vll86\" (UniqueName: \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.629529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.629509 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.692314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.692275 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vll86\" (UniqueName: \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\") pod \"payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:12.727737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.727704 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:12.769301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:12.769265 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:13.120821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.120791 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:01:13.121258 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:01:13.121063 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55efd5dc_3fa5_448e_87e3_6510d7e039a2.slice/crio-726495ef0467b4298c0655abfcda31a9f528df5ba11b25cb4939fb5f44ed9bfa WatchSource:0}: Error finding container 726495ef0467b4298c0655abfcda31a9f528df5ba11b25cb4939fb5f44ed9bfa: Status 404 returned error can't find the container with id 726495ef0467b4298c0655abfcda31a9f528df5ba11b25cb4939fb5f44ed9bfa Nov 03 18:01:13.176976 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.176946 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:01:13.177273 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:01:13.177252 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3fa0a0b_f2aa_448d_b433_5e01e68e9a1f.slice/crio-2ee6e4a98b49f2835691117c95e8f01d33676e8bc19522cdb391fcc6b1cd1aa6 WatchSource:0}: Error finding container 2ee6e4a98b49f2835691117c95e8f01d33676e8bc19522cdb391fcc6b1cd1aa6: Status 404 returned error can't find the container with id 2ee6e4a98b49f2835691117c95e8f01d33676e8bc19522cdb391fcc6b1cd1aa6 Nov 03 18:01:13.443481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.443443 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" event={"ID":"55efd5dc-3fa5-448e-87e3-6510d7e039a2","Type":"ContainerStarted","Data":"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db"} Nov 03 18:01:13.443720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.443505 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" event={"ID":"55efd5dc-3fa5-448e-87e3-6510d7e039a2","Type":"ContainerStarted","Data":"726495ef0467b4298c0655abfcda31a9f528df5ba11b25cb4939fb5f44ed9bfa"} Nov 03 18:01:13.443831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.443794 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:13.445769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.445742 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" event={"ID":"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f","Type":"ContainerStarted","Data":"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0"} Nov 03 18:01:13.445888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.445792 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" event={"ID":"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f","Type":"ContainerStarted","Data":"2ee6e4a98b49f2835691117c95e8f01d33676e8bc19522cdb391fcc6b1cd1aa6"} Nov 03 18:01:13.445888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.445823 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:13.447366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.447341 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerStarted","Data":"51499cb40fe65ba5617ec120e921293c48e2599e1e8117c350dc88a4e10ba982"} Nov 03 18:01:13.569076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.569027 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" podStartSLOduration=2.5690157559999998 podStartE2EDuration="2.569015756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:13.566680698 +0000 UTC m=+1653.395764300" watchObservedRunningTime="2026-11-03 18:01:13.569015756 +0000 UTC m=+1653.398099358" Nov 03 18:01:13.707217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.707103 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" podStartSLOduration=1.707076357 podStartE2EDuration="1.707076357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:13.694265767 +0000 UTC m=+1653.523349372" watchObservedRunningTime="2026-11-03 18:01:13.707076357 +0000 UTC m=+1653.536159965" Nov 03 18:01:13.847631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:13.842349 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podStartSLOduration=3.842325858 podStartE2EDuration="3.842325858s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:13.828618393 +0000 UTC m=+1653.657702009" watchObservedRunningTime="2026-11-03 18:01:13.842325858 +0000 UTC m=+1653.671409461" Nov 03 18:01:14.374591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.374541 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:14.557663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.555069 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:14.687665 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:14.683798 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:01:14.731902 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.731868 4980 patch_prober.go:29] interesting pod/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" start-of-body= Nov 03 18:01:14.732058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.731932 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.82:15021/healthz/ready\": dial tcp 10.132.0.82:15021: connect: connection refused" Nov 03 18:01:14.866785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.866757 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 18:01:14.953820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.953791 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.953987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.953831 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954012 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954137 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954268 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954248 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:14.954342 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954284 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954342 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954315 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954342 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954385 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:14.954466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954444 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954491 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\") pod \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\" (UID: \"0199e352-db8a-4f9f-a88e-1edcbf8eb738\") " Nov 03 18:01:14.954654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954532 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:14.954964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954935 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:14.955046 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.954964 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:14.955614 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.955550 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:14.955614 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.955597 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:14.955614 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.955613 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:14.955832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.955627 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:14.955832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.955642 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:14.956282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.956254 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "kube-api-access-qt4s7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:14.956368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.956348 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:01:14.956745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.956728 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:14.956856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:14.956835 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy" pod "0199e352-db8a-4f9f-a88e-1edcbf8eb738" (UID: "0199e352-db8a-4f9f-a88e-1edcbf8eb738"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:15.057992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.057942 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:15.057992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.057982 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt4s7\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-kube-api-access-qt4s7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:15.057992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.057994 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:15.057992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.058003 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0199e352-db8a-4f9f-a88e-1edcbf8eb738-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.546782 4980 generic.go:356] "Generic (PLEG): container finished" podID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerID="feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f" exitCode=0 Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.548299 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerDied","Data":"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f"} Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.548410 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.548437 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" event={"ID":"0199e352-db8a-4f9f-a88e-1edcbf8eb738","Type":"ContainerDied","Data":"3132ea5df2b93f6ff0ea708df6269245af7c63c2f0922af70776100bbfac83bc"} Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.548464 4980 scope.go:122] "RemoveContainer" containerID="feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f" Nov 03 18:01:15.549030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.548679 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv" Nov 03 18:01:15.557535 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.557505 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:01:15.568374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.568356 4980 scope.go:122] "RemoveContainer" containerID="e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba" Nov 03 18:01:15.578984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.578966 4980 scope.go:122] "RemoveContainer" containerID="feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f" Nov 03 18:01:15.579237 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:15.579216 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f\": container with ID starting with feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f not found: ID does not exist" containerID="feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f" Nov 03 18:01:15.579302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.579246 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f"} err="failed to get container status \"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f\": rpc error: code = NotFound desc = could not find container \"feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f\": container with ID starting with feb486b833be6cfa685eac606fc2b79c722bd5ba85cc45729cbea8ec8251b30f not found: ID does not exist" Nov 03 18:01:15.579302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.579265 4980 scope.go:122] "RemoveContainer" containerID="e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba" Nov 03 18:01:15.579520 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:15.579500 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba\": container with ID starting with e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba not found: ID does not exist" containerID="e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba" Nov 03 18:01:15.579592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.579524 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba"} err="failed to get container status \"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba\": rpc error: code = NotFound desc = could not find container \"e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba\": container with ID starting with e2f14551b7e565915582d5fd129336f6bf775d75e5a7637c1ffbe0dd08c03dba not found: ID does not exist" Nov 03 18:01:15.724064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.724027 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 18:01:15.774314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:15.774282 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-1c1ea78c-openshift-default-95494844c-tq7nv"] Nov 03 18:01:16.297492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:16.297461 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" path="/var/lib/kubelet/pods/0199e352-db8a-4f9f-a88e-1edcbf8eb738/volumes" Nov 03 18:01:22.889386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.889137 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:01:22.895514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.895411 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.895770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.895745 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.895978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.895964 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.896060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.896050 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.896635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.896614 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.896762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.896750 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0199e352-db8a-4f9f-a88e-1edcbf8eb738" containerName="istio-proxy" Nov 03 18:01:22.904960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.904928 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:01:22.905208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.905193 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:01:22.912840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.912807 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:22.934868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.930950 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:01:22.934868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.931086 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:22.981154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.981126 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-394741e0\"" Nov 03 18:01:22.992452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.992413 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mn4vp\" (UniqueName: \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:22.992721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:22.992694 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.029061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.029033 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-394741e0-dockercfg-l228l\"" Nov 03 18:01:23.094181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.094145 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.094327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.094208 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.094388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.094368 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m978s\" (UniqueName: \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.094462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.094440 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mn4vp\" (UniqueName: \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.094972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.094947 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.157826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.157752 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mn4vp\" (UniqueName: \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\") pod \"payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.195397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.195367 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m978s\" (UniqueName: \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.195603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.195582 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.196263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.196237 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.258669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.258638 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m978s\" (UniqueName: \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\") pod \"payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.273453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.273426 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.282641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.282613 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.670067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.670040 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:01:23.670240 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:01:23.670217 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5228e2_f3c8_453c_98a0_e136b4b13cf6.slice/crio-761df4f816789016cd2cb220e0c7a150052ec1079be91dc1a062a0854857ae7d WatchSource:0}: Error finding container 761df4f816789016cd2cb220e0c7a150052ec1079be91dc1a062a0854857ae7d: Status 404 returned error can't find the container with id 761df4f816789016cd2cb220e0c7a150052ec1079be91dc1a062a0854857ae7d Nov 03 18:01:23.724902 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:01:23.724646 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf064fc2c_b8af_4f89_82ec_c7ac36728720.slice/crio-1bccf83a4fcaca562c5a1c63a1cdcbcbe26a8f39fb4107c66e7ce722c9e85a45 WatchSource:0}: Error finding container 1bccf83a4fcaca562c5a1c63a1cdcbcbe26a8f39fb4107c66e7ce722c9e85a45: Status 404 returned error can't find the container with id 1bccf83a4fcaca562c5a1c63a1cdcbcbe26a8f39fb4107c66e7ce722c9e85a45 Nov 03 18:01:23.725828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.725797 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:01:23.859830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.858959 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" event={"ID":"f064fc2c-b8af-4f89-82ec-c7ac36728720","Type":"ContainerStarted","Data":"68513976c896bae89460f79088edfbe6d18104edf0bb9152658bf4ff36a2961f"} Nov 03 18:01:23.859830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.859029 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" event={"ID":"f064fc2c-b8af-4f89-82ec-c7ac36728720","Type":"ContainerStarted","Data":"1bccf83a4fcaca562c5a1c63a1cdcbcbe26a8f39fb4107c66e7ce722c9e85a45"} Nov 03 18:01:23.859830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.859095 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:23.861147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.861116 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" event={"ID":"ac5228e2-f3c8-453c-98a0-e136b4b13cf6","Type":"ContainerStarted","Data":"ccdf81b04efd1064e4a5820fa1c7b92c6986489cb4352cff9397fdc057886448"} Nov 03 18:01:23.861282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.861200 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:23.861282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:23.861225 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" event={"ID":"ac5228e2-f3c8-453c-98a0-e136b4b13cf6","Type":"ContainerStarted","Data":"761df4f816789016cd2cb220e0c7a150052ec1079be91dc1a062a0854857ae7d"} Nov 03 18:01:24.121597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:24.120938 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" podStartSLOduration=2.120915825 podStartE2EDuration="2.120915825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:23.978755832 +0000 UTC m=+1663.807839449" watchObservedRunningTime="2026-11-03 18:01:24.120915825 +0000 UTC m=+1663.949999429" Nov 03 18:01:24.132555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:24.130833 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" podStartSLOduration=2.130812374 podStartE2EDuration="2.130812374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:01:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:01:24.123869209 +0000 UTC m=+1663.952952812" watchObservedRunningTime="2026-11-03 18:01:24.130812374 +0000 UTC m=+1663.959895976" Nov 03 18:01:24.521361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:24.521327 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:01:24.543926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:24.543901 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:01:34.872283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:34.872251 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:01:34.872872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:34.872328 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:01:44.523667 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:44.523619 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:01:44.789341 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:44.789253 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 18:01:44.789558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:44.789511 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" containerID="cri-o://2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3" gracePeriod=30 Nov 03 18:01:50.113606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.113540 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:01:50.212458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212410 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.212692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212506 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.212692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212626 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.212841 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212725 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.212841 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212766 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:50.212841 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212790 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.213000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212852 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:50.213000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212912 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.213000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.212956 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.213151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.213014 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.213151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.213070 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\") pod \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\" (UID: \"891a371e-7bec-49a8-9300-0d3bc0c9ec57\") " Nov 03 18:01:50.213151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.213136 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:01:50.213363 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.213322 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:50.213621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.213508 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:50.214050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.214026 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.214153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.214055 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.214153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.214070 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.214153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.214083 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.214153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.214097 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.215179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.215157 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:01:50.215280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.215201 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:01:50.215427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.215407 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:50.215905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.215884 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2" pod "891a371e-7bec-49a8-9300-0d3bc0c9ec57" (UID: "891a371e-7bec-49a8-9300-0d3bc0c9ec57"). InnerVolumeSpecName "kube-api-access-7drm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:01:50.406029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.405977 4980 generic.go:356] "Generic (PLEG): container finished" podID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerID="2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3" exitCode=0 Nov 03 18:01:50.408394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.408268 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.408394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.408301 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.408394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.408326 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7drm2\" (UniqueName: \"kubernetes.io/projected/891a371e-7bec-49a8-9300-0d3bc0c9ec57-kube-api-access-7drm2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.408394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.408343 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/891a371e-7bec-49a8-9300-0d3bc0c9ec57-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:01:50.411744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.411706 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" Nov 03 18:01:50.420599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.420555 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerDied","Data":"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3"} Nov 03 18:01:50.420806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.420790 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k" event={"ID":"891a371e-7bec-49a8-9300-0d3bc0c9ec57","Type":"ContainerDied","Data":"8aa33d2080e656927b4c0af8b58996669ddbc924df0c594b65ba70fa311313d0"} Nov 03 18:01:50.420937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.420921 4980 scope.go:122] "RemoveContainer" containerID="2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3" Nov 03 18:01:50.438968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.438947 4980 scope.go:122] "RemoveContainer" containerID="966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab" Nov 03 18:01:50.449265 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.449241 4980 scope.go:122] "RemoveContainer" containerID="2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3" Nov 03 18:01:50.449532 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:50.449510 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3\": container with ID starting with 2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3 not found: ID does not exist" containerID="2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3" Nov 03 18:01:50.449644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.449546 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3"} err="failed to get container status \"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3\": rpc error: code = NotFound desc = could not find container \"2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3\": container with ID starting with 2dd1d2cffd1899d8ee1e8fd0fcea4a093d8d3048f1bf27ef2d2dd581f43cc2a3 not found: ID does not exist" Nov 03 18:01:50.449644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.449613 4980 scope.go:122] "RemoveContainer" containerID="966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab" Nov 03 18:01:50.449879 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:01:50.449859 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab\": container with ID starting with 966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab not found: ID does not exist" containerID="966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab" Nov 03 18:01:50.449937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.449887 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab"} err="failed to get container status \"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab\": rpc error: code = NotFound desc = could not find container \"966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab\": container with ID starting with 966c5915429fd6d705944873feb4646c1286b85506bcc8d9e3a522232bdbb5ab not found: ID does not exist" Nov 03 18:01:50.586263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.586230 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 18:01:50.591556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:50.591532 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-aa500779-openshift-default-599b5bf9f7-mch6k"] Nov 03 18:01:52.295409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:01:52.295377 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" path="/var/lib/kubelet/pods/891a371e-7bec-49a8-9300-0d3bc0c9ec57/volumes" Nov 03 18:02:11.607715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.607684 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:02:11.608688 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608666 4980 cpu_manager.go:414] "containerMap: removing container" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.608688 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608688 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.608865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608735 4980 cpu_manager.go:414] "containerMap: removing container" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.608865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608743 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.608982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608968 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.609041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.608993 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="891a371e-7bec-49a8-9300-0d3bc0c9ec57" containerName="istio-proxy" Nov 03 18:02:11.611805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.611783 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.718823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.718791 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:02:11.744321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744280 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744345 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744434 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744487 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744765 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744747 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744864 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.744962 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.744947 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.745031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.745012 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.745103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.745087 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnkxx\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.846964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.846929 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnkxx\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.846994 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847017 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847042 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847068 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847363 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847224 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847363 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847294 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847380 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847410 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847534 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847663 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847760 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.847831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.847784 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.848283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.848255 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.849841 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.849810 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.850127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.850106 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.910500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.910434 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.910641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.910594 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnkxx\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4wzlm\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:11.925125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:11.925100 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:12.366049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.366018 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:02:12.366525 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:12.366504 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda240db05_8056_412e_baeb_a22cea0e066c.slice/crio-e0aef5bb7b1188682f902897d667153b82479dcbfc8b1257ca8758b78a55e2b0 WatchSource:0}: Error finding container e0aef5bb7b1188682f902897d667153b82479dcbfc8b1257ca8758b78a55e2b0: Status 404 returned error can't find the container with id e0aef5bb7b1188682f902897d667153b82479dcbfc8b1257ca8758b78a55e2b0 Nov 03 18:02:12.369181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.369140 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:12.369277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.369230 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:12.369317 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.369273 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:12.603475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.603433 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" event={"ID":"a240db05-8056-412e-baeb-a22cea0e066c","Type":"ContainerStarted","Data":"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8"} Nov 03 18:02:12.603744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.603502 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" event={"ID":"a240db05-8056-412e-baeb-a22cea0e066c","Type":"ContainerStarted","Data":"e0aef5bb7b1188682f902897d667153b82479dcbfc8b1257ca8758b78a55e2b0"} Nov 03 18:02:12.728889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.728837 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podStartSLOduration=1.728822939 podStartE2EDuration="1.728822939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:12.726955993 +0000 UTC m=+1712.556039595" watchObservedRunningTime="2026-11-03 18:02:12.728822939 +0000 UTC m=+1712.557906540" Nov 03 18:02:12.925756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:12.925722 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:13.927556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:13.927519 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:13.928076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:13.927593 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:14.522873 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:02:14.522834 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:02:14.810256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:14.810177 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:02:14.813996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:14.813971 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:14.874748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:14.874054 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-e75517-openshift-default-dockercfg-fhbsd\"" Nov 03 18:02:15.084677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.008809 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.085366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.085328 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.137767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.137723 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.148815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.148781 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtvk4\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.163811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.163767 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.210708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.210660 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.225941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.225897 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.235758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.235712 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.272601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.260764 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.272601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.116073 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:02:15.272601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.210928 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:15.272601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.261620 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:15.498650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.386478 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.499056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.499017 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.514653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.499276 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtvk4\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.514937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.397028 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.515032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.513718 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.515120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.515046 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.522725 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.522687 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.523039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.523020 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.523211 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.523196 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.523372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.523358 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.523673 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.523651 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.524372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.524344 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.525233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.525211 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.534505 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.534469 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.545609 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.545554 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.572270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.572197 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.593956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.593857 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtvk4\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.597371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.597264 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\") pod \"e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.730464 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.729269 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:15.926283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.926249 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:15.926460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:15.926303 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:16.204234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.204198 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:02:16.204814 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:16.204792 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c049c75_fe72_4c2c_b675_cb5a1ee3c70f.slice/crio-01d0651e770939ed79dc6be17f2647d891465ca4b31839105bf78d3fa00a07a1 WatchSource:0}: Error finding container 01d0651e770939ed79dc6be17f2647d891465ca4b31839105bf78d3fa00a07a1: Status 404 returned error can't find the container with id 01d0651e770939ed79dc6be17f2647d891465ca4b31839105bf78d3fa00a07a1 Nov 03 18:02:16.207999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.207962 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:16.208110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.208042 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:16.208110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.208088 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:16.702065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.702020 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerStarted","Data":"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169"} Nov 03 18:02:16.702284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.702091 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerStarted","Data":"01d0651e770939ed79dc6be17f2647d891465ca4b31839105bf78d3fa00a07a1"} Nov 03 18:02:16.751592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.749797 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:16.893602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.891600 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podStartSLOduration=2.8915567639999997 podStartE2EDuration="2.891556764s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:16.845308039 +0000 UTC m=+1716.674391643" watchObservedRunningTime="2026-11-03 18:02:16.891556764 +0000 UTC m=+1716.720640366" Nov 03 18:02:16.927345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.926698 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:16.927345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:16.926933 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:17.012015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:17.011923 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 18:02:17.012212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:17.012112 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" containerID="cri-o://c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807" gracePeriod=30 Nov 03 18:02:17.735479 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:17.735451 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:17.925982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:17.925942 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:17.926129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:17.926008 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:18.744803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:18.744754 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:18.746547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:18.746524 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:02:18.926534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:18.926504 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:18.926754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:18.926557 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:19.926115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:19.926076 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:19.926527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:19.926136 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:20.926237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:20.926202 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:02:20.926767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:20.926278 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:02:22.077617 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.077533 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:22.078106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.077992 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:22.079014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.078991 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:02:22.316221 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.316196 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:02:22.398422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398381 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398489 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398616 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398688 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398731 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398841 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.398908 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398894 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.399001 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.398986 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.399044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.399024 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\") pod \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\" (UID: \"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f\") " Nov 03 18:02:22.401799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.400922 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:02:22.401799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.401462 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:02:22.401799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.402510 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:02:22.401799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.402788 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:02:22.409565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.404877 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:02:22.418821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.415064 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:02:22.418821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.415093 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:02:22.418821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.415186 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "kube-api-access-gs2f6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:02:22.418821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.417030 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo" pod "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" (UID: "5d6a6372-3795-41df-b1b0-b8a91b6f9c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502884 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502917 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502934 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502950 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502963 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502978 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.502991 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.503006 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.505686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.503019 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs2f6\" (UniqueName: \"kubernetes.io/projected/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f-kube-api-access-gs2f6\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:02:22.783414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.783378 4980 generic.go:356] "Generic (PLEG): container finished" podID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerID="c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807" exitCode=0 Nov 03 18:02:22.783620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.783431 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" Nov 03 18:02:22.783620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.783457 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerDied","Data":"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807"} Nov 03 18:02:22.783620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.783513 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq" event={"ID":"5d6a6372-3795-41df-b1b0-b8a91b6f9c0f","Type":"ContainerDied","Data":"973fb8d4ffc0368fb5979cb11c42b89ec78ea5a0561225b4a33090b5d7effa7f"} Nov 03 18:02:22.783620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.783536 4980 scope.go:122] "RemoveContainer" containerID="c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807" Nov 03 18:02:22.807903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.807873 4980 scope.go:122] "RemoveContainer" containerID="936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb" Nov 03 18:02:22.818530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.818504 4980 scope.go:122] "RemoveContainer" containerID="c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807" Nov 03 18:02:22.818819 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:02:22.818798 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807\": container with ID starting with c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807 not found: ID does not exist" containerID="c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807" Nov 03 18:02:22.818892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.818829 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807"} err="failed to get container status \"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807\": rpc error: code = NotFound desc = could not find container \"c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807\": container with ID starting with c11baea07941d779906aed05d2730ba3ab1da9b750a063785123d2fed738c807 not found: ID does not exist" Nov 03 18:02:22.818892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.818848 4980 scope.go:122] "RemoveContainer" containerID="936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb" Nov 03 18:02:22.819098 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:02:22.819084 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb\": container with ID starting with 936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb not found: ID does not exist" containerID="936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb" Nov 03 18:02:22.819143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.819101 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb"} err="failed to get container status \"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb\": rpc error: code = NotFound desc = could not find container \"936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb\": container with ID starting with 936b96e5c18ce1861b393e06b0175b80f8a2ae9f04289e85f8e0c9bdf0bd9beb not found: ID does not exist" Nov 03 18:02:22.955759 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:22.955724 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 18:02:23.005882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.005847 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-12a446f9-openshift-default-65bbc88fb5-xd7cq"] Nov 03 18:02:23.725661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.723870 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:02:23.726233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.725728 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:23.726233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.725752 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:23.726233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.726054 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:23.726233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.726077 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:23.737605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.730740 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:23.810603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.799244 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-e75517-dockercfg-8gtpb\"" Nov 03 18:02:23.811008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.810981 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-e75517\"" Nov 03 18:02:23.863432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.863394 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:02:23.944624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.937730 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:02:23.991651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.958693 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:23.991651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:23.958725 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" containerName="istio-proxy" Nov 03 18:02:24.132598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.095894 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr7d4\" (UniqueName: \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.132598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.096033 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.160599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.133152 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.203813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.203778 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr7d4\" (UniqueName: \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.224731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.224326 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.226561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.226532 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.386596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.375451 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.386596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.375921 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvlp5\" (UniqueName: \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.386596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.378309 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr7d4\" (UniqueName: \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\") pod \"payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.386596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.385114 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d6a6372-3795-41df-b1b0-b8a91b6f9c0f" path="/var/lib/kubelet/pods/5d6a6372-3795-41df-b1b0-b8a91b6f9c0f/volumes" Nov 03 18:02:24.386596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.385623 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:02:24.443842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.443804 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:24.477873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.477837 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.478101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.478076 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvlp5\" (UniqueName: \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.478695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.478672 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.541420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.541386 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvlp5\" (UniqueName: \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\") pod \"payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.552816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.552786 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:24.835378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.835343 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:02:24.835956 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:24.835928 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67767ae8_7eab_4610_968f_2511f3569102.slice/crio-d98dc0c84f425a83d435f9f6aabfc3e267274f94f839184085b350e21696fb84 WatchSource:0}: Error finding container d98dc0c84f425a83d435f9f6aabfc3e267274f94f839184085b350e21696fb84: Status 404 returned error can't find the container with id d98dc0c84f425a83d435f9f6aabfc3e267274f94f839184085b350e21696fb84 Nov 03 18:02:24.955252 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:24.955220 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf047702e_2c8d_4ca3_9526_7067055dc854.slice/crio-36f7f5cdabf319387be8f5eca56a279567c73981e61d51f7716ecee5545d8f5b WatchSource:0}: Error finding container 36f7f5cdabf319387be8f5eca56a279567c73981e61d51f7716ecee5545d8f5b: Status 404 returned error can't find the container with id 36f7f5cdabf319387be8f5eca56a279567c73981e61d51f7716ecee5545d8f5b Nov 03 18:02:24.955396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:24.955258 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:02:25.156774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.154273 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" event={"ID":"67767ae8-7eab-4610-968f-2511f3569102","Type":"ContainerStarted","Data":"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01"} Nov 03 18:02:25.156774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.154339 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" event={"ID":"67767ae8-7eab-4610-968f-2511f3569102","Type":"ContainerStarted","Data":"d98dc0c84f425a83d435f9f6aabfc3e267274f94f839184085b350e21696fb84"} Nov 03 18:02:25.156774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.155592 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:25.172693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.171176 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" event={"ID":"f047702e-2c8d-4ca3-9526-7067055dc854","Type":"ContainerStarted","Data":"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02"} Nov 03 18:02:25.172693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.171223 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" event={"ID":"f047702e-2c8d-4ca3-9526-7067055dc854","Type":"ContainerStarted","Data":"36f7f5cdabf319387be8f5eca56a279567c73981e61d51f7716ecee5545d8f5b"} Nov 03 18:02:25.172693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.171245 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:25.292637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.292558 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" podStartSLOduration=2.292541797 podStartE2EDuration="2.292541797s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:25.29172385 +0000 UTC m=+1725.120807453" watchObservedRunningTime="2026-11-03 18:02:25.292541797 +0000 UTC m=+1725.121625398" Nov 03 18:02:25.526468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:25.519597 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" podStartSLOduration=2.519553703 podStartE2EDuration="2.519553703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:25.433854688 +0000 UTC m=+1725.262938292" watchObservedRunningTime="2026-11-03 18:02:25.519553703 +0000 UTC m=+1725.348637306" Nov 03 18:02:29.236042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.235756 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:02:29.243310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.243280 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298357 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298426 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298475 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298502 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298563 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jr5lv\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298666 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.298988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298953 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.299046 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.298995 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.299131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.299104 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.299801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.299778 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6ed3e95c-openshift-default-dockercfg-rk4jm\"" Nov 03 18:02:29.355406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.355363 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:02:29.416312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.414318 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jr5lv\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.416312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.414441 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.416312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.414641 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.421741 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.421698 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.422208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.422180 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.422446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.422424 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.422930 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.422904 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.423130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.423109 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.423297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.423277 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.423425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.423409 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.426825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.426803 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.427622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.427558 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.443596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.439068 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.448671 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.444589 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.448671 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.444839 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.448671 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.445048 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.492088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.492015 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.493320 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.493264 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jr5lv\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:29.573972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:29.560527 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:30.152362 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:30.152331 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:02:30.153456 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:30.153433 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91fd53ed_22db_4403_b2a8_ce80b2c393be.slice/crio-276c6340cd3ada69310b4ba98e502858600548fcef7177ab7b75cc7d086358a9 WatchSource:0}: Error finding container 276c6340cd3ada69310b4ba98e502858600548fcef7177ab7b75cc7d086358a9: Status 404 returned error can't find the container with id 276c6340cd3ada69310b4ba98e502858600548fcef7177ab7b75cc7d086358a9 Nov 03 18:02:30.156198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:30.156168 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:30.156284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:30.156231 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:30.156284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:30.156261 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:30.223638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:30.220767 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerStarted","Data":"276c6340cd3ada69310b4ba98e502858600548fcef7177ab7b75cc7d086358a9"} Nov 03 18:02:31.229396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:31.229354 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerStarted","Data":"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297"} Nov 03 18:02:31.406996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:31.406944 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podStartSLOduration=2.406929244 podStartE2EDuration="2.406929244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:31.402188839 +0000 UTC m=+1731.231272441" watchObservedRunningTime="2026-11-03 18:02:31.406929244 +0000 UTC m=+1731.236012846" Nov 03 18:02:31.572717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:31.569838 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:31.616612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:31.613916 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:32.237069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:32.237043 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:32.238091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:32.238067 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:02:36.183244 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:36.183218 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:02:37.190775 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:37.190745 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:02:38.075306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.075276 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:02:38.079748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.079729 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.119381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.119356 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:02:38.131669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.131640 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.131813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.131794 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdwc8\" (UniqueName: \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.134035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.134012 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6ed3e95c\"" Nov 03 18:02:38.203602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.190723 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6ed3e95c-dockercfg-6s6mz\"" Nov 03 18:02:38.278619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.271507 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.385890 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.385763 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdwc8\" (UniqueName: \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.485663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.462945 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.641639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.639241 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:02:38.641639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.641083 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdwc8\" (UniqueName: \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\") pod \"payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.645918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.645899 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:02:38.646102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.646000 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.695397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.695366 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:38.759738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.759711 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.760050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.760033 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8s8l\" (UniqueName: \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.861472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.861428 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.861717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.861704 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8s8l\" (UniqueName: \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.862349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.862322 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.925472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.925396 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8s8l\" (UniqueName: \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\") pod \"payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:38.972543 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:38.972507 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:39.084270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.084241 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:02:39.088182 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:39.088150 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2676795d_e35e_4387_93fc_85411a98d35f.slice/crio-086b45fe7ac919fcf3f16d87ec13c26a6f192cae7977e53e758221a2dd9b21b7 WatchSource:0}: Error finding container 086b45fe7ac919fcf3f16d87ec13c26a6f192cae7977e53e758221a2dd9b21b7: Status 404 returned error can't find the container with id 086b45fe7ac919fcf3f16d87ec13c26a6f192cae7977e53e758221a2dd9b21b7 Nov 03 18:02:39.361586 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.361542 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:02:39.362332 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:02:39.362308 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa3314f3_0313_40e8_b3ba_c6fe6c7732c3.slice/crio-d1436da164c3cc07d5b1e793b685951fb32e0b25fd875dabf04f788648e0b81e WatchSource:0}: Error finding container d1436da164c3cc07d5b1e793b685951fb32e0b25fd875dabf04f788648e0b81e: Status 404 returned error can't find the container with id d1436da164c3cc07d5b1e793b685951fb32e0b25fd875dabf04f788648e0b81e Nov 03 18:02:39.654564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.654475 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" event={"ID":"2676795d-e35e-4387-93fc-85411a98d35f","Type":"ContainerStarted","Data":"f530de6bcda477c8a024cc4379c1f571fb4d624ff25fc27cfd6f1a87c9493029"} Nov 03 18:02:39.654564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.654536 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" event={"ID":"2676795d-e35e-4387-93fc-85411a98d35f","Type":"ContainerStarted","Data":"086b45fe7ac919fcf3f16d87ec13c26a6f192cae7977e53e758221a2dd9b21b7"} Nov 03 18:02:39.654741 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.654583 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:39.656128 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.656104 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" event={"ID":"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3","Type":"ContainerStarted","Data":"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d"} Nov 03 18:02:39.656218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.656144 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" event={"ID":"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3","Type":"ContainerStarted","Data":"d1436da164c3cc07d5b1e793b685951fb32e0b25fd875dabf04f788648e0b81e"} Nov 03 18:02:39.656272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.656237 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:39.777512 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.777453 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" podStartSLOduration=1.777433657 podStartE2EDuration="1.777433657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:39.775614085 +0000 UTC m=+1739.604697682" watchObservedRunningTime="2026-11-03 18:02:39.777433657 +0000 UTC m=+1739.606517261" Nov 03 18:02:39.914665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:39.914434 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" podStartSLOduration=1.914414595 podStartE2EDuration="1.914414595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:39.913703411 +0000 UTC m=+1739.742787016" watchObservedRunningTime="2026-11-03 18:02:39.914414595 +0000 UTC m=+1739.743498198" Nov 03 18:02:44.523196 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:02:44.523159 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:02:50.729843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:50.728788 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:02:50.758597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:50.747744 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:02:51.351913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.351133 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:02:51.461406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.461361 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:51.573696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.573649 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:51.603092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.602972 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:02:51.710742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.705672 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:51.710742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.707392 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 18:02:51.722719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:51.722678 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.268892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.268838 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.308837 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.308789 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.356421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.356383 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:02:52.496607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.494113 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.520027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.519932 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:02:52.520311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.520295 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:02:52.537471 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.529637 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 18:02:52.574866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.574823 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.575512 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.575488 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:02:52.616321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.616274 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:02:52.617269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.617232 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:02:52.720717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.720365 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:02:52.753713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.723789 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 18:02:52.753713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.724615 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:02:52.753713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.735053 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:02:52.849659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.842118 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:02:52.849659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.842198 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:02:52.849659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.842257 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:02:52.849659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.848434 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:02:52.924936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.924885 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:02:52.925193 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:52.924987 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:02:56.817789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.817754 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:02:56.844090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.844061 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893335 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893486 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893682 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893660 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893732 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rtsc\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893794 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893766 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893798 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.893927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893903 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.894062 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.893958 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.894127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.894071 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.955011 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.954958 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:02:56.995630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995595 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995725 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995772 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rtsc\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995813 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995860 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995891 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.995989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995962 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996200 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.995987 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996200 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996084 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996200 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996061 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996360 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996272 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996360 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996315 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996592 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.996919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.996734 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.998480 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.998456 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:56.998745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:56.998722 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:57.059703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:57.059661 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:57.059856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:57.059757 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rtsc\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:57.175187 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:57.175096 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:58.000642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:58.000531 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:02:58.004351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:58.004309 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:58.004485 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:58.004399 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:58.004485 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:58.004447 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:02:58.106329 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:58.106289 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" event={"ID":"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d","Type":"ContainerStarted","Data":"8e31a2ab1c9f5f72d3a928da6c148810add96f77a27ce522bc62eeda3d9fb2a6"} Nov 03 18:02:59.116413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:59.116373 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" event={"ID":"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d","Type":"ContainerStarted","Data":"699c780335e11acca2d2d73a73e7614814dc48cf5898632c4e21e78c178c6180"} Nov 03 18:02:59.177597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:59.176973 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:02:59.181085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:59.180753 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:02:59.181085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:59.180826 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:02:59.353796 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:02:59.353743 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podStartSLOduration=3.353728229 podStartE2EDuration="3.353728229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:02:59.349622586 +0000 UTC m=+1759.178706187" watchObservedRunningTime="2026-11-03 18:02:59.353728229 +0000 UTC m=+1759.182811831" Nov 03 18:03:00.183066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:00.183019 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:03:00.183544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:00.183083 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:03:01.178597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:01.177940 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:03:01.178597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:01.178011 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:03:01.626755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:01.626710 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:03:01.627160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:01.626779 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:02.187638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:02.186996 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:03:02.187638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:02.187076 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:03:03.178032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:03.177987 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:03:03.178692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:03.178658 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:03:04.181730 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:04.181675 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" start-of-body= Nov 03 18:03:04.182169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:04.181761 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.114:15021/healthz/ready\": dial tcp 10.132.0.114:15021: connect: connection refused" Nov 03 18:03:05.181141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:05.181112 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:03:05.181659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:05.181624 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:03:05.182394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:05.182364 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:03:07.219697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:07.219667 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:03:07.220330 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:07.219728 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:11.873010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:11.872963 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:03:11.984764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:11.983999 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:11.990771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:11.990722 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:03:12.103662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.100266 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.114261 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.115546 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.115743 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.115835 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.115857 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.116014 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.116169 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.116386 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx66f\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.116507 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.116605 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.155186 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.217220 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.217266 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.217353 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218045 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218094 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218138 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218176 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218207 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx66f\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218374 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218444 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218504 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218533 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218643 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218767 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfsm9\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.218907 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219020 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219037 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219112 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219155 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219183 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219258 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.219344 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.220006 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.220273 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.220868 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.221224 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.234192 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.234221 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.282698 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx66f\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.313235 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\") pod \"e2e-ait-394741e0-openshift-default-545f5658c5-crdnv\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.320925 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.320985 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321091 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321141 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321170 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321382 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321417 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321517 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321542 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfsm9\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321695 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321738 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321776 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321821 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321919 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.321934 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322041 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322064 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phh6w\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322129 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322171 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322501 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322611 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322563 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.322700 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.324157 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.324586 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.384431 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.384635 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfsm9\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\") pod \"maas-default-gateway-openshift-default-5df48fcc99-2ggk9\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.423730 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.423813 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.423846 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.423894 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.423917 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424049 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424121 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424160 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.431287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424218 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424286 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phh6w\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424510 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424563 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424917 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.424943 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.426334 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.434112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.427513 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.487430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.487388 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.488535 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.488512 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phh6w\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\") pod \"maas-default-gateway-openshift-default-5df48fcc99-nm6mn\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.507855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.507831 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:12.554907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.554880 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:12.761159 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:12.610769 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:13.387279 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:03:13.387239 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71271103_783a_4333_97a1_c8a25dba560a.slice/crio-9933b6691bb1d634b6cc7face2a13542f92b9c829244b8c8cd1d0a509a27781c WatchSource:0}: Error finding container 9933b6691bb1d634b6cc7face2a13542f92b9c829244b8c8cd1d0a509a27781c: Status 404 returned error can't find the container with id 9933b6691bb1d634b6cc7face2a13542f92b9c829244b8c8cd1d0a509a27781c Nov 03 18:03:13.387853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.387830 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:03:13.389966 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.389933 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.390071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.389996 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.390071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.390027 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.420191 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:03:13.420161 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d23304_b75e_47d1_b37d_60a707459bac.slice/crio-8ea9063b2590272fc46e34ea63391a433bcd4402c0ecc550b0e749cdb03f5e0a WatchSource:0}: Error finding container 8ea9063b2590272fc46e34ea63391a433bcd4402c0ecc550b0e749cdb03f5e0a: Status 404 returned error can't find the container with id 8ea9063b2590272fc46e34ea63391a433bcd4402c0ecc550b0e749cdb03f5e0a Nov 03 18:03:13.423169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.423134 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.423274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.423208 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.423274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.423257 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.437679 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:03:13.437650 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a4e16c_3777_4d4a_8c05_15c543064878.slice/crio-254c607336805010b63dd25745e30be24cf7fd3d6728af5f7cbe94e6845078c9 WatchSource:0}: Error finding container 254c607336805010b63dd25745e30be24cf7fd3d6728af5f7cbe94e6845078c9: Status 404 returned error can't find the container with id 254c607336805010b63dd25745e30be24cf7fd3d6728af5f7cbe94e6845078c9 Nov 03 18:03:13.441677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.441614 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:03:13.451009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.450820 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:03:13.457548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.457516 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.457676 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.457629 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.457748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.457692 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.525810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.525266 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" event={"ID":"71271103-783a-4333-97a1-c8a25dba560a","Type":"ContainerStarted","Data":"9933b6691bb1d634b6cc7face2a13542f92b9c829244b8c8cd1d0a509a27781c"} Nov 03 18:03:13.527912 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.527887 4980 generic.go:356] "Generic (PLEG): container finished" podID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerID="51499cb40fe65ba5617ec120e921293c48e2599e1e8117c350dc88a4e10ba982" exitCode=0 Nov 03 18:03:13.528007 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.527982 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerDied","Data":"51499cb40fe65ba5617ec120e921293c48e2599e1e8117c350dc88a4e10ba982"} Nov 03 18:03:13.528531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.528513 4980 scope.go:122] "RemoveContainer" containerID="51499cb40fe65ba5617ec120e921293c48e2599e1e8117c350dc88a4e10ba982" Nov 03 18:03:13.530639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.530605 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.530743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.530692 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.530743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.530737 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.534113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.533992 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" event={"ID":"66d23304-b75e-47d1-b37d-60a707459bac","Type":"ContainerStarted","Data":"8ea9063b2590272fc46e34ea63391a433bcd4402c0ecc550b0e749cdb03f5e0a"} Nov 03 18:03:13.538754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.538721 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" event={"ID":"57a4e16c-3777-4d4a-8c05-15c543064878","Type":"ContainerStarted","Data":"254c607336805010b63dd25745e30be24cf7fd3d6728af5f7cbe94e6845078c9"} Nov 03 18:03:13.540662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.540635 4980 generic.go:356] "Generic (PLEG): container finished" podID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerID="d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c" exitCode=0 Nov 03 18:03:13.540779 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.540668 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerDied","Data":"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c"} Nov 03 18:03:13.541192 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.541172 4980 scope.go:122] "RemoveContainer" containerID="d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c" Nov 03 18:03:13.547528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.547432 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.547723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.547563 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:13.547723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:13.547637 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:03:14.553367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.553305 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerStarted","Data":"cd29504e96a153a0aab7ca77acfd1946d063aecf5e3957bce93188ba0e5c30ad"} Nov 03 18:03:14.555517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.555357 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" event={"ID":"66d23304-b75e-47d1-b37d-60a707459bac","Type":"ContainerStarted","Data":"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591"} Nov 03 18:03:14.557456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.557426 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" event={"ID":"57a4e16c-3777-4d4a-8c05-15c543064878","Type":"ContainerStarted","Data":"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c"} Nov 03 18:03:14.559526 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:14.559500 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:03:14.559697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.559682 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerStarted","Data":"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89"} Nov 03 18:03:14.561365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.561343 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" event={"ID":"71271103-783a-4333-97a1-c8a25dba560a","Type":"ContainerStarted","Data":"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393"} Nov 03 18:03:14.619599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.618951 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:14.620820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.620536 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:14.620820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.620624 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:14.896643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:14.879371 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podStartSLOduration=2.879351362 podStartE2EDuration="2.879351362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:14.878424185 +0000 UTC m=+1774.707507790" watchObservedRunningTime="2026-11-03 18:03:14.879351362 +0000 UTC m=+1774.708434966" Nov 03 18:03:15.109600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.099654 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:03:15.109600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.099784 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:15.109600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.099834 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:15.208654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.206802 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podStartSLOduration=3.206778428 podStartE2EDuration="3.206778428s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:15.177699902 +0000 UTC m=+1775.006783502" watchObservedRunningTime="2026-11-03 18:03:15.206778428 +0000 UTC m=+1775.035862031" Nov 03 18:03:15.393599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.373942 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:03:15.393599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.392707 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:15.393599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.392788 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:15.509657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.508952 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:15.513596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.512793 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:15.513596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.512856 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:15.550596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.550037 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:03:15.586599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.571964 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:15.590593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.587401 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:15.590593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.587459 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:15.635596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.630010 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:15.635596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.630095 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:15.885212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.843738 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:03:15.885212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.858699 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:15.946563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.935393 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:15.946563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:15.945719 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.110599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.108041 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:16.110599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.108109 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:16.311820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.311781 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.371476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.371437 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:16.371742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.371712 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:16.554621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.553040 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:16.554621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.553117 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:16.588653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.561157 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:16.588653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.561213 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:16.664756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.661066 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.664756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.661523 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:16.664756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.661672 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:16.698598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.687726 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.806598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.806178 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.852662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.851223 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:03:16.852662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.851736 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:03:16.874607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.874249 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:16.879607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.875147 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:03:16.996280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.980810 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:16.996280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:16.983305 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:03:17.100595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.098998 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": context deadline exceeded" Nov 03 18:03:17.213594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.212965 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:17.213594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.213023 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:17.460716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.343628 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:17.460716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.344077 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:03:17.460716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.420673 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podStartSLOduration=6.4206497240000004 podStartE2EDuration="6.420649724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:03:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:03:15.305616208 +0000 UTC m=+1775.134699810" watchObservedRunningTime="2026-11-03 18:03:17.420649724 +0000 UTC m=+1777.249733327" Nov 03 18:03:17.491810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.491720 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:17.493604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.492095 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:17.526199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.526129 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:17.533037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.533001 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:17.533298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.533268 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:17.575628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.574259 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:17.575628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.574318 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:17.616663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.613976 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:17.616663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.614037 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:17.698165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.678551 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:17.698165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.681401 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:03:17.698859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.698792 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:17.702064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.702031 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:03:17.818665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.809554 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:17.818665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.810158 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:03:17.861977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.859878 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:17.877722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.877673 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:17.942598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.941743 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:18.015598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:17.992697 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:03:18.288596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.275422 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:18.288596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.275487 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:18.377695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.363661 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:18.377887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.377678 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:18.435929 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.435890 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:18.436208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.436179 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:18.539411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.539369 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:18.762597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.757147 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:18.762597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.692745 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:18.774673 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.555630 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 18:03:18.788110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.775985 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:18.788398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.788366 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:18.788534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.711348 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 18:03:18.834606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.833687 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:18.916612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.914762 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:18.916612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.914854 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:18.951540 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.951484 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:18.952291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:18.952165 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:19.042838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.036270 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:03:19.217391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.217333 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:19.225982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.220702 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:19.225982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.220785 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:19.372047 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:19.371955 4980 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.006s" Nov 03 18:03:19.372437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.372397 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:03:19.372655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.372641 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:03:19.433961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.433925 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:19.434279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.434240 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:19.483963 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.436053 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:19.483963 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.436123 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:20.021599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:19.986900 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:20.112302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.069601 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.112302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.069661 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:20.112302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.070620 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.112302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.070666 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:20.112302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.073194 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 18:03:20.147297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.147256 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/health\": context deadline exceeded" Nov 03 18:03:20.161729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.149638 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 18:03:20.161729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.149904 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 18:03:20.161729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.152693 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.161729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.152736 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:20.207640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.192142 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:20.207640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.192407 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:20.308789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.289086 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:03:20.308789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.307720 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.308789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.307787 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:20.322254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.322201 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:03:20.371167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.371123 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.371463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.371416 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:20.544086 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.544038 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:20.594322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.573862 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.594322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.573948 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:20.706278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.706240 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.706587 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.706531 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:20.798259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.798105 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:20.798259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:20.798174 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:21.072621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.072554 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 18:03:21.099596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.096681 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 18:03:21.140243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.140203 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:21.140510 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.140485 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:21.189492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.189454 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:03:21.189797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.189776 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:03:21.199994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.195411 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:03:21.208769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.208717 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 18:03:21.235446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.219308 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:21.269178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.269134 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:21.269783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.269744 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 18:03:21.305954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.298891 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:21.467310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.322949 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:21.564597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.550973 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:21.564597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.551038 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:21.602091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.602054 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:21.602376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.602341 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:21.631913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.631873 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:21.632209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.632174 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:21.669949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.669910 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:21.693600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.670164 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:21.697400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:21.550731 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.101897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.101053 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.101897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.101773 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:22.101897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.101822 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:22.102584 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.101895 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.235536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.235487 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 18:03:22.261426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.246698 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.280752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.280691 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.281473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.281441 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 18:03:22.281994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.281967 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": context deadline exceeded" Nov 03 18:03:22.505641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.478889 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:03:22.505641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.484771 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:22.505641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.484826 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:22.505641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.484909 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.527605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.508840 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:22.527605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.508902 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:22.597596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.576067 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:22.607666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.601721 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:22.743597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.701997 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:22.915460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.839811 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.015602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:22.957527 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:23.104229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.104189 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.104844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.104810 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:23.113079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.113036 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.139090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.117230 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.236692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.236652 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.256608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.249583 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded" Nov 03 18:03:23.366611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.322224 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.366611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.322638 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.521658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.461960 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.607597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.494825 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:03:23.607597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.494867 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.607597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.522331 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.607597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.597216 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:23.716313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.716275 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.744677 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.943645 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.795802 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.944137 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.796750 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:23.944603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:23.944194 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:24.128598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.118143 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:24.128598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.118217 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:24.169598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.144972 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": context deadline exceeded" Nov 03 18:03:24.211643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.200789 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.243597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.239965 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.243597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.237884 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 18:03:24.305067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.304023 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.419650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.361670 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:24.419650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.362070 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.419650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.372933 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:24.419650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.372984 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:24.516057 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.515965 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:24.516313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.516283 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:24.551595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.551284 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 03 18:03:24.558590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.555980 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:24.558590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.556038 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:24.604804 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.604737 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.605220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.605092 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:24.616107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.616072 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:24.616391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.616346 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:24.952723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:24.952672 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:25.107005 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.106969 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body= Nov 03 18:03:25.107251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.107223 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" Nov 03 18:03:25.149174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.149040 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": context deadline exceeded" Nov 03 18:03:25.182430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.181543 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:25.216355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.216253 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 18:03:25.278601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.276666 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": context deadline exceeded" Nov 03 18:03:25.304718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.304659 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:25.362323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.362274 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:25.363013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.362980 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:25.370943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.370913 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body= Nov 03 18:03:25.371153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.371128 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" Nov 03 18:03:25.511252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.511143 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:25.511551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.511505 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:25.557044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.557003 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:25.557298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.557268 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:25.614777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.614713 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:25.615853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.615459 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" start-of-body= Nov 03 18:03:25.616670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.616626 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.115:15021/healthz/ready\": dial tcp 10.132.0.115:15021: connect: connection refused" Nov 03 18:03:25.616901 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.616864 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:25.695834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.695783 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:03:25.745024 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:25.744996 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:03:26.133591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.132062 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:03:26.188057 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.187681 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:03:26.202592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.201090 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" podUID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" containerName="payload-pre-processing" containerID="cri-o://d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0" gracePeriod=25 Nov 03 18:03:26.202592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.201259 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" podUID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" containerName="payload-processing" containerID="cri-o://43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db" gracePeriod=25 Nov 03 18:03:26.218503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.217708 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:26.282525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.280592 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:26.342513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.327659 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:26.384105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.382669 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:03:26.414810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.414787 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:03:26.506317 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.505256 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:03:26.517487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.517459 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:03:26.531604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.520225 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:03:26.557471 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.556341 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:03:26.560039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.559936 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:26.560039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.559988 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:26.563560 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.562648 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:03:26.585926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.585894 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:26.586150 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.586126 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:26.608129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.608099 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:03:26.621967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.620810 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:03:26.623734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.623670 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:03:26.638639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.634110 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:03:26.638639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.635145 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:26.638639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.635724 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:26.638639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.637188 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:03:26.750417 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.750373 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 18:03:26.750924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.750892 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" containerID="cri-o://e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914" gracePeriod=30 Nov 03 18:03:26.757098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.757072 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 18:03:26.757456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.757428 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" containerID="cri-o://27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349" gracePeriod=30 Nov 03 18:03:26.968717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:26.968671 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:03:27.060111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.060081 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:03:27.147458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.147425 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\" (UniqueName: \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\") pod \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " Nov 03 18:03:27.147735 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.147551 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\") pod \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\" (UID: \"55efd5dc-3fa5-448e-87e3-6510d7e039a2\") " Nov 03 18:03:27.147813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.147764 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\" (UniqueName: \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\") pod \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " Nov 03 18:03:27.147873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.147848 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\") pod \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\" (UID: \"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f\") " Nov 03 18:03:27.148239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.148209 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume" pod "55efd5dc-3fa5-448e-87e3-6510d7e039a2" (UID: "55efd5dc-3fa5-448e-87e3-6510d7e039a2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:27.148495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.148467 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume" pod "f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" (UID: "f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:27.148759 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.148737 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55efd5dc-3fa5-448e-87e3-6510d7e039a2-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:27.148831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.148766 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:27.151501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.151474 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86" pod "f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" (UID: "f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f"). InnerVolumeSpecName "kube-api-access-vll86". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:27.152698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.152670 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm" pod "55efd5dc-3fa5-448e-87e3-6510d7e039a2" (UID: "55efd5dc-3fa5-448e-87e3-6510d7e039a2"). InnerVolumeSpecName "kube-api-access-5h9tm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:27.220921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.220816 4980 generic.go:356] "Generic (PLEG): container finished" podID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" containerID="d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0" exitCode=0 Nov 03 18:03:27.220921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.220881 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" event={"ID":"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f","Type":"ContainerDied","Data":"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0"} Nov 03 18:03:27.221455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.220862 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" Nov 03 18:03:27.221455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.221022 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n" event={"ID":"f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f","Type":"ContainerDied","Data":"2ee6e4a98b49f2835691117c95e8f01d33676e8bc19522cdb391fcc6b1cd1aa6"} Nov 03 18:03:27.221455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.221062 4980 scope.go:122] "RemoveContainer" containerID="d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0" Nov 03 18:03:27.223207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.223173 4980 generic.go:356] "Generic (PLEG): container finished" podID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" containerID="43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db" exitCode=0 Nov 03 18:03:27.224303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.224283 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" Nov 03 18:03:27.224432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.224403 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" event={"ID":"55efd5dc-3fa5-448e-87e3-6510d7e039a2","Type":"ContainerDied","Data":"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db"} Nov 03 18:03:27.224510 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.224494 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5" event={"ID":"55efd5dc-3fa5-448e-87e3-6510d7e039a2","Type":"ContainerDied","Data":"726495ef0467b4298c0655abfcda31a9f528df5ba11b25cb4939fb5f44ed9bfa"} Nov 03 18:03:27.239107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.239085 4980 scope.go:122] "RemoveContainer" containerID="d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0" Nov 03 18:03:27.239614 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:27.239583 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0\": container with ID starting with d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0 not found: ID does not exist" containerID="d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0" Nov 03 18:03:27.239681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.239629 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0"} err="failed to get container status \"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0\": rpc error: code = NotFound desc = could not find container \"d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0\": container with ID starting with d714c0f95b5002b95de08932033b4b1c40cf22205ba3a38ead6341f8e7d0aae0 not found: ID does not exist" Nov 03 18:03:27.239681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.239661 4980 scope.go:122] "RemoveContainer" containerID="43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db" Nov 03 18:03:27.251016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.250989 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vll86\" (UniqueName: \"kubernetes.io/projected/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f-kube-api-access-vll86\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:27.251137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.251030 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h9tm\" (UniqueName: \"kubernetes.io/projected/55efd5dc-3fa5-448e-87e3-6510d7e039a2-kube-api-access-5h9tm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:27.252442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.252417 4980 scope.go:122] "RemoveContainer" containerID="43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db" Nov 03 18:03:27.252906 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:27.252879 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db\": container with ID starting with 43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db not found: ID does not exist" containerID="43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db" Nov 03 18:03:27.252989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.252921 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db"} err="failed to get container status \"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db\": rpc error: code = NotFound desc = could not find container \"43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db\": container with ID starting with 43927f29bcca53eaa3f1697a6ba2db69baede53cbc194a05e325ca711df951db not found: ID does not exist" Nov 03 18:03:27.509096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.509001 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:27.509391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.509360 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-2ggk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" start-of-body= Nov 03 18:03:27.509534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.509415 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.116:15021/healthz/ready\": dial tcp 10.132.0.116:15021: connect: connection refused" Nov 03 18:03:27.556734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.556702 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:27.556989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.556958 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" start-of-body= Nov 03 18:03:27.557110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:27.557018 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": dial tcp 10.132.0.117:15021: connect: connection refused" Nov 03 18:03:28.180338 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.180305 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:03:28.184782 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.184756 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3e138c04-77f445b4d5-ft2w5"] Nov 03 18:03:28.244689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.244651 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:03:28.309706 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.309669 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" path="/var/lib/kubelet/pods/55efd5dc-3fa5-448e-87e3-6510d7e039a2/volumes" Nov 03 18:03:28.360786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.360722 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:03:28.390641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.386835 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:03:28.395336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:28.395302 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-3e138c04-565f6748bc-fn55n"] Nov 03 18:03:29.393419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.392891 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:29.397601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.397546 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:03:29.632956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.632919 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-nm6mn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:03:29.633164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.632980 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.117:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:03:29.727440 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.727409 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:29.728792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:29.728765 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:03:30.552600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:30.547900 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" path="/var/lib/kubelet/pods/f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f/volumes" Nov 03 18:03:31.516235 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:31.516194 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:03:31.516392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:31.516263 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:03:32.109595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.109554 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 18:03:32.130160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130129 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130213 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130330 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130606 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130537 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:32.130728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130650 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130667 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.130728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130722 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130831 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.130887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130860 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130912 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.130989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130934 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.130989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.130949 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\") pod \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\" (UID: \"3c2d53fa-70fe-4707-a617-6b6ccc87321c\") " Nov 03 18:03:32.131160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.131068 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.132085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.132062 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.132191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.132090 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.132191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.132105 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.132191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.132118 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.132191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.132131 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.133353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.133324 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:03:32.133451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.133405 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.133720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.133700 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:32.133790 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.133734 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd" pod "3c2d53fa-70fe-4707-a617-6b6ccc87321c" (UID: "3c2d53fa-70fe-4707-a617-6b6ccc87321c"). InnerVolumeSpecName "kube-api-access-ft7wd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:32.140534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.140516 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 18:03:32.232991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.232951 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233003 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233109 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233184 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233244 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.233335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233321 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233359 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233416 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233393 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233450 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233486 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.233558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233504 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\") pod \"5829e122-cc6e-4a48-9468-868a1eefdf37\" (UID: \"5829e122-cc6e-4a48-9468-868a1eefdf37\") " Nov 03 18:03:32.233805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233763 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.233805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233779 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.233935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.233806 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:32.234619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234595 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234626 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234641 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5829e122-cc6e-4a48-9468-868a1eefdf37-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234654 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234668 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234683 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft7wd\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-kube-api-access-ft7wd\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234698 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234712 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c2d53fa-70fe-4707-a617-6b6ccc87321c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.234854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.234726 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.235664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.235641 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:03:32.235823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.235801 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:32.236077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.236058 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "kube-api-access-jmmx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:32.236138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.236098 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy" pod "5829e122-cc6e-4a48-9468-868a1eefdf37" (UID: "5829e122-cc6e-4a48-9468-868a1eefdf37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:03:32.336686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.336595 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmmx7\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-kube-api-access-jmmx7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.336686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.336628 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.336686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.336637 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.336686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.336645 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5829e122-cc6e-4a48-9468-868a1eefdf37-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:32.472439 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.472403 4980 generic.go:356] "Generic (PLEG): container finished" podID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerID="e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914" exitCode=0 Nov 03 18:03:32.472667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.472483 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" Nov 03 18:03:32.472667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.472499 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" event={"ID":"3c2d53fa-70fe-4707-a617-6b6ccc87321c","Type":"ContainerDied","Data":"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914"} Nov 03 18:03:32.473138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.473117 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg" event={"ID":"3c2d53fa-70fe-4707-a617-6b6ccc87321c","Type":"ContainerDied","Data":"065c0ba90798029d9582909a39b6de83412e03961072eec815dc3e877a82f72f"} Nov 03 18:03:32.473229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.473180 4980 scope.go:122] "RemoveContainer" containerID="e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914" Nov 03 18:03:32.474736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.474507 4980 generic.go:356] "Generic (PLEG): container finished" podID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerID="27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349" exitCode=0 Nov 03 18:03:32.474736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.474549 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" event={"ID":"5829e122-cc6e-4a48-9468-868a1eefdf37","Type":"ContainerDied","Data":"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349"} Nov 03 18:03:32.474736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.474589 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" Nov 03 18:03:32.474736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.474617 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474" event={"ID":"5829e122-cc6e-4a48-9468-868a1eefdf37","Type":"ContainerDied","Data":"163870aae496072f24c1bda622b1132d0223afebd343cf57611bc12cc58f0802"} Nov 03 18:03:32.508954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.508938 4980 scope.go:122] "RemoveContainer" containerID="e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914" Nov 03 18:03:32.509262 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:32.509233 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914\": container with ID starting with e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914 not found: ID does not exist" containerID="e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914" Nov 03 18:03:32.509340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.509270 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914"} err="failed to get container status \"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914\": rpc error: code = NotFound desc = could not find container \"e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914\": container with ID starting with e25e9315f0fa7d3e98d6a2867a8ce143bb0466f06377421b6c9e2b85708a7914 not found: ID does not exist" Nov 03 18:03:32.509340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.509290 4980 scope.go:122] "RemoveContainer" containerID="27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349" Nov 03 18:03:32.520080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.520061 4980 scope.go:122] "RemoveContainer" containerID="27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349" Nov 03 18:03:32.520409 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:32.520387 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349\": container with ID starting with 27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349 not found: ID does not exist" containerID="27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349" Nov 03 18:03:32.520475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.520417 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349"} err="failed to get container status \"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349\": rpc error: code = NotFound desc = could not find container \"27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349\": container with ID starting with 27ea04e867535a3b4610b776cfc25ebcc666caa86184b1a19a91d53cf3be8349 not found: ID does not exist" Nov 03 18:03:32.658836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.658747 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 18:03:32.691845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.691802 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-fk474"] Nov 03 18:03:32.810017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.809984 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 18:03:32.815543 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:32.815515 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-d25mg"] Nov 03 18:03:33.246420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.246373 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" podUID="f064fc2c-b8af-4f89-82ec-c7ac36728720" containerName="payload-processing" containerID="cri-o://68513976c896bae89460f79088edfbe6d18104edf0bb9152658bf4ff36a2961f" gracePeriod=25 Nov 03 18:03:33.362030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.361982 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" podUID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" containerName="payload-pre-processing" containerID="cri-o://ccdf81b04efd1064e4a5820fa1c7b92c6986489cb4352cff9397fdc057886448" gracePeriod=25 Nov 03 18:03:33.490692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.490565 4980 generic.go:356] "Generic (PLEG): container finished" podID="f064fc2c-b8af-4f89-82ec-c7ac36728720" containerID="68513976c896bae89460f79088edfbe6d18104edf0bb9152658bf4ff36a2961f" exitCode=0 Nov 03 18:03:33.490874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.490630 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" event={"ID":"f064fc2c-b8af-4f89-82ec-c7ac36728720","Type":"ContainerDied","Data":"68513976c896bae89460f79088edfbe6d18104edf0bb9152658bf4ff36a2961f"} Nov 03 18:03:33.492657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.492634 4980 generic.go:356] "Generic (PLEG): container finished" podID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" containerID="ccdf81b04efd1064e4a5820fa1c7b92c6986489cb4352cff9397fdc057886448" exitCode=0 Nov 03 18:03:33.492786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.492704 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" event={"ID":"ac5228e2-f3c8-453c-98a0-e136b4b13cf6","Type":"ContainerDied","Data":"ccdf81b04efd1064e4a5820fa1c7b92c6986489cb4352cff9397fdc057886448"} Nov 03 18:03:33.568356 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.568324 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:03:33.662343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.662323 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\" (UniqueName: \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\") pod \"f064fc2c-b8af-4f89-82ec-c7ac36728720\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " Nov 03 18:03:33.662473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.662459 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\") pod \"f064fc2c-b8af-4f89-82ec-c7ac36728720\" (UID: \"f064fc2c-b8af-4f89-82ec-c7ac36728720\") " Nov 03 18:03:33.662888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.662863 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume" pod "f064fc2c-b8af-4f89-82ec-c7ac36728720" (UID: "f064fc2c-b8af-4f89-82ec-c7ac36728720"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:33.663275 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.663260 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f064fc2c-b8af-4f89-82ec-c7ac36728720-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:33.665017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.664992 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s" pod "f064fc2c-b8af-4f89-82ec-c7ac36728720" (UID: "f064fc2c-b8af-4f89-82ec-c7ac36728720"). InnerVolumeSpecName "kube-api-access-m978s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:33.670388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.670361 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:03:33.763720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.763689 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\" (UniqueName: \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\") pod \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " Nov 03 18:03:33.763891 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.763749 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\") pod \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\" (UID: \"ac5228e2-f3c8-453c-98a0-e136b4b13cf6\") " Nov 03 18:03:33.764210 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.764178 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume" pod "ac5228e2-f3c8-453c-98a0-e136b4b13cf6" (UID: "ac5228e2-f3c8-453c-98a0-e136b4b13cf6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:03:33.764536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.764509 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m978s\" (UniqueName: \"kubernetes.io/projected/f064fc2c-b8af-4f89-82ec-c7ac36728720-kube-api-access-m978s\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:33.764536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.764535 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:33.766113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.766081 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp" pod "ac5228e2-f3c8-453c-98a0-e136b4b13cf6" (UID: "ac5228e2-f3c8-453c-98a0-e136b4b13cf6"). InnerVolumeSpecName "kube-api-access-mn4vp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:03:33.866334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:33.866289 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mn4vp\" (UniqueName: \"kubernetes.io/projected/ac5228e2-f3c8-453c-98a0-e136b4b13cf6-kube-api-access-mn4vp\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:03:34.295649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.295613 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" path="/var/lib/kubelet/pods/3c2d53fa-70fe-4707-a617-6b6ccc87321c/volumes" Nov 03 18:03:34.296199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.296181 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" path="/var/lib/kubelet/pods/5829e122-cc6e-4a48-9468-868a1eefdf37/volumes" Nov 03 18:03:34.501562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.501534 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" Nov 03 18:03:34.501562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.501556 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl" event={"ID":"f064fc2c-b8af-4f89-82ec-c7ac36728720","Type":"ContainerDied","Data":"1bccf83a4fcaca562c5a1c63a1cdcbcbe26a8f39fb4107c66e7ce722c9e85a45"} Nov 03 18:03:34.501841 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.501627 4980 scope.go:122] "RemoveContainer" containerID="68513976c896bae89460f79088edfbe6d18104edf0bb9152658bf4ff36a2961f" Nov 03 18:03:34.503237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.503215 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" event={"ID":"ac5228e2-f3c8-453c-98a0-e136b4b13cf6","Type":"ContainerDied","Data":"761df4f816789016cd2cb220e0c7a150052ec1079be91dc1a062a0854857ae7d"} Nov 03 18:03:34.504027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.503219 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9" Nov 03 18:03:34.513037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.513020 4980 scope.go:122] "RemoveContainer" containerID="ccdf81b04efd1064e4a5820fa1c7b92c6986489cb4352cff9397fdc057886448" Nov 03 18:03:34.672422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.672343 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:03:34.678911 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.678885 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-394741e0-57d886bdcc-9lzcl"] Nov 03 18:03:34.792599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.792555 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:03:34.796537 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:34.796512 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-394741e0-5c48dfcc9-85dm9"] Nov 03 18:03:36.296530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:36.296501 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" path="/var/lib/kubelet/pods/ac5228e2-f3c8-453c-98a0-e136b4b13cf6/volumes" Nov 03 18:03:36.297145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:36.297127 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f064fc2c-b8af-4f89-82ec-c7ac36728720" path="/var/lib/kubelet/pods/f064fc2c-b8af-4f89-82ec-c7ac36728720/volumes" Nov 03 18:03:40.560805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:40.560776 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:03:40.561442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:40.561421 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:03:40.566847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:40.566826 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:03:40.566957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:03:40.566842 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:03:44.585928 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:03:44.585885 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:04:00.743616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.743587 4980 generic.go:356] "Generic (PLEG): container finished" podID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerID="480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169" exitCode=0 Nov 03 18:04:00.743616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.743607 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerDied","Data":"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169"} Nov 03 18:04:00.744119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.744103 4980 scope.go:122] "RemoveContainer" containerID="480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169" Nov 03 18:04:00.745971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.745941 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:00.746044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.746025 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:00.746094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:00.746077 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:01.753538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:01.753498 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerStarted","Data":"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3"} Nov 03 18:04:02.729423 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:02.729390 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:04:02.730762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:02.730643 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 18:04:02.730889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:02.730796 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 18:04:03.730729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:03.730689 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 18:04:03.731146 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:03.730758 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 18:04:03.744938 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:03.744908 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:04:04.731009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:04.730970 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 18:04:04.731659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:04.731617 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 18:04:05.744558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:05.741959 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 18:04:05.744558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:05.742079 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 18:04:06.736678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:06.736642 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" start-of-body= Nov 03 18:04:06.744633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:06.744587 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.108:15021/healthz/ready\": dial tcp 10.132.0.108:15021: connect: connection refused" Nov 03 18:04:07.866593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:07.866339 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:04:07.869241 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:07.869214 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:04:08.551925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:08.551884 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:04:08.552174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:08.552144 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" containerID="cri-o://e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89" gracePeriod=30 Nov 03 18:04:09.109652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.109614 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:04:09.140065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.140028 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:04:09.140752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.140717 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" containerID="cri-o://cd29504e96a153a0aab7ca77acfd1946d063aecf5e3957bce93188ba0e5c30ad" gracePeriod=30 Nov 03 18:04:09.141884 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.141853 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" containerID="cri-o://63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591" gracePeriod=30 Nov 03 18:04:09.330591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.330536 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:04:09.331207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:09.331174 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" containerID="cri-o://699c780335e11acca2d2d73a73e7614814dc48cf5898632c4e21e78c178c6180" gracePeriod=30 Nov 03 18:04:11.508913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.508876 4980 patch_prober.go:29] interesting pod/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:11.509316 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.508958 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:11.624298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.624224 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:11.624298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.624302 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:11.638306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.638276 4980 patch_prober.go:29] interesting pod/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:11.638424 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:11.638328 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:13.859980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:13.859953 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:04:13.985598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:13.977328 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:13.985598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:13.977446 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:13.985598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:13.977481 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:13.997145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:13.997081 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010406 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010522 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010656 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010742 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010795 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010871 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.010973 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\") pod \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\" (UID: \"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6\") " Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.014177 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.014206 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.017221 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.017632 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.017888 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.031058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.018817 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.068783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.068714 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:14.069667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.069619 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "kube-api-access-f7xpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.075864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.075750 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token" pod "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" (UID: "d0d47cb0-cb62-4ae4-aa0a-348421ca18d6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.116967 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117006 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117021 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117036 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117050 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7xpm\" (UniqueName: \"kubernetes.io/projected/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-kube-api-access-f7xpm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117066 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.117322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.117080 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.235785 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:14.235731 4980 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d23304_b75e_47d1_b37d_60a707459bac.slice/crio-63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d23304_b75e_47d1_b37d_60a707459bac.slice/crio-conmon-63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591.scope\": RecentStats: unable to find data in memory cache]" Nov 03 18:04:14.356971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.356943 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:04:14.421821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.421785 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.421967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.421848 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.421967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.421915 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422018 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422068 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.422113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422078 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422166 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.422292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422259 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422289 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422346 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422391 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\") pod \"66d23304-b75e-47d1-b37d-60a707459bac\" (UID: \"66d23304-b75e-47d1-b37d-60a707459bac\") " Nov 03 18:04:14.422468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422404 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.422770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422745 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:14.424105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.422972 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.424888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.424860 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.424998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.424893 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.424998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.424910 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.424998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.424925 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.424998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.424940 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66d23304-b75e-47d1-b37d-60a707459bac-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.425843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.425810 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.436217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.435416 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:14.436217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.435669 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "kube-api-access-wx66f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.437345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.437318 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token" pod "66d23304-b75e-47d1-b37d-60a707459bac" (UID: "66d23304-b75e-47d1-b37d-60a707459bac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.500663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.499667 4980 generic.go:356] "Generic (PLEG): container finished" podID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerID="cd29504e96a153a0aab7ca77acfd1946d063aecf5e3957bce93188ba0e5c30ad" exitCode=0 Nov 03 18:04:14.500663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.499735 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerDied","Data":"cd29504e96a153a0aab7ca77acfd1946d063aecf5e3957bce93188ba0e5c30ad"} Nov 03 18:04:14.500663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.499955 4980 scope.go:122] "RemoveContainer" containerID="51499cb40fe65ba5617ec120e921293c48e2599e1e8117c350dc88a4e10ba982" Nov 03 18:04:14.506834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.506806 4980 generic.go:356] "Generic (PLEG): container finished" podID="66d23304-b75e-47d1-b37d-60a707459bac" containerID="63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591" exitCode=0 Nov 03 18:04:14.506954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.506890 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" event={"ID":"66d23304-b75e-47d1-b37d-60a707459bac","Type":"ContainerDied","Data":"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591"} Nov 03 18:04:14.506954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.506929 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" event={"ID":"66d23304-b75e-47d1-b37d-60a707459bac","Type":"ContainerDied","Data":"8ea9063b2590272fc46e34ea63391a433bcd4402c0ecc550b0e749cdb03f5e0a"} Nov 03 18:04:14.507054 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.506992 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv" Nov 03 18:04:14.518675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.518643 4980 scope.go:122] "RemoveContainer" containerID="63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591" Nov 03 18:04:14.521857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.521823 4980 generic.go:356] "Generic (PLEG): container finished" podID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerID="e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89" exitCode=0 Nov 03 18:04:14.521962 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.521902 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" Nov 03 18:04:14.522072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.522051 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerDied","Data":"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89"} Nov 03 18:04:14.522131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.522099 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv" event={"ID":"d0d47cb0-cb62-4ae4-aa0a-348421ca18d6","Type":"ContainerDied","Data":"dee54f7b3e57ee41120ca93161c1978da1b172377378e145b6fe5c9419497197"} Nov 03 18:04:14.524711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.524688 4980 generic.go:356] "Generic (PLEG): container finished" podID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerID="699c780335e11acca2d2d73a73e7614814dc48cf5898632c4e21e78c178c6180" exitCode=0 Nov 03 18:04:14.524807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.524734 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" event={"ID":"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d","Type":"ContainerDied","Data":"699c780335e11acca2d2d73a73e7614814dc48cf5898632c4e21e78c178c6180"} Nov 03 18:04:14.525301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.525277 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:04:14.526518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.526470 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66d23304-b75e-47d1-b37d-60a707459bac-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.526518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.526495 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.526518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.526506 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66d23304-b75e-47d1-b37d-60a707459bac-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.526518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.526515 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx66f\" (UniqueName: \"kubernetes.io/projected/66d23304-b75e-47d1-b37d-60a707459bac-kube-api-access-wx66f\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.540897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.540876 4980 scope.go:122] "RemoveContainer" containerID="63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591" Nov 03 18:04:14.541194 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:14.541171 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591\": container with ID starting with 63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591 not found: ID does not exist" containerID="63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591" Nov 03 18:04:14.541286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.541205 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591"} err="failed to get container status \"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591\": rpc error: code = NotFound desc = could not find container \"63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591\": container with ID starting with 63488e7e791d4dfca9709602cead3a6d81938a7d40149af33d9dde3df9da0591 not found: ID does not exist" Nov 03 18:04:14.541286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.541230 4980 scope.go:122] "RemoveContainer" containerID="e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89" Nov 03 18:04:14.542291 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:14.542264 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:04:14.554535 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.554502 4980 scope.go:122] "RemoveContainer" containerID="d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c" Nov 03 18:04:14.567401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.567374 4980 scope.go:122] "RemoveContainer" containerID="e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89" Nov 03 18:04:14.567747 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:14.567726 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89\": container with ID starting with e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89 not found: ID does not exist" containerID="e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89" Nov 03 18:04:14.567833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.567757 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89"} err="failed to get container status \"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89\": rpc error: code = NotFound desc = could not find container \"e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89\": container with ID starting with e9748c00fd8201dbf46174b9f47f8c36e92b6029e531f3912ceb3e82d5e59f89 not found: ID does not exist" Nov 03 18:04:14.567833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.567782 4980 scope.go:122] "RemoveContainer" containerID="d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c" Nov 03 18:04:14.568061 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:14.568044 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c\": container with ID starting with d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c not found: ID does not exist" containerID="d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c" Nov 03 18:04:14.568127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.568066 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c"} err="failed to get container status \"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c\": rpc error: code = NotFound desc = could not find container \"d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c\": container with ID starting with d83cdc4d19db1197461673aa63190efede4a880b09c8642efa51658f00ef2e9c not found: ID does not exist" Nov 03 18:04:14.627012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.626985 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627096 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627118 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627167 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627265 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627325 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627392 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627527 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:14.627607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627597 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.627724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627664 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\") pod \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\" (UID: \"fe87c6e6-03da-45a0-9cae-2854d65e15ef\") " Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.627808 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628330 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628421 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628639 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628663 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628679 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628694 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.628857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.628754 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.631026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.630989 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:14.631143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.631122 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.631833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.631804 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "kube-api-access-w9tv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.632977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.632951 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy" pod "fe87c6e6-03da-45a0-9cae-2854d65e15ef" (UID: "fe87c6e6-03da-45a0-9cae-2854d65e15ef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.703548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.703514 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:04:14.730454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.730417 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.730454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.730456 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.730721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.730470 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.730721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.730485 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe87c6e6-03da-45a0-9cae-2854d65e15ef-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.730721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.730500 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9tv7\" (UniqueName: \"kubernetes.io/projected/fe87c6e6-03da-45a0-9cae-2854d65e15ef-kube-api-access-w9tv7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.816147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.816111 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:04:14.830002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.829598 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 18:04:14.830002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.829831 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" containerID="cri-o://c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3" gracePeriod=30 Nov 03 18:04:14.831441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.831400 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.831636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.831463 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.831636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.831608 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.831792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.831640 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.832089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.831934 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.832089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.832002 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.832089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.832054 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.832089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.832073 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.832089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.832094 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\") pod \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\" (UID: \"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d\") " Nov 03 18:04:14.834969 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.833111 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:14.834969 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.833488 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.836183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.836128 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.836891 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.836863 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.839167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.838992 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.839949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.839906 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:14.840305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.840176 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.840403 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.840309 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:14.840842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.840815 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc" pod "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" (UID: "b0390d5c-53b5-4f5f-8b12-308fce3b5c1d"). InnerVolumeSpecName "kube-api-access-4rtsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:14.843533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.843504 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 18:04:14.843982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.843933 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" containerID="cri-o://1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54" gracePeriod=30 Nov 03 18:04:14.850243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.850209 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 18:04:14.850503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.850474 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" containerID="cri-o://c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4" gracePeriod=30 Nov 03 18:04:14.856771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.856741 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 18:04:14.857021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.856994 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" containerID="cri-o://826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69" gracePeriod=30 Nov 03 18:04:14.867610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.867207 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-crdnv"] Nov 03 18:04:14.934552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934517 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934545 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934557 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934581 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934594 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934602 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934611 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934619 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rtsc\" (UniqueName: \"kubernetes.io/projected/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-kube-api-access-4rtsc\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.934889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.934630 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:14.990462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.988455 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:04:14.994100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:14.994077 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-3e138c04-openshift-default-54d4cfcd4c-xk5sv"] Nov 03 18:04:15.535237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.535206 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" Nov 03 18:04:15.535401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.535250 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz" event={"ID":"b0390d5c-53b5-4f5f-8b12-308fce3b5c1d","Type":"ContainerDied","Data":"8e31a2ab1c9f5f72d3a928da6c148810add96f77a27ce522bc62eeda3d9fb2a6"} Nov 03 18:04:15.535401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.535310 4980 scope.go:122] "RemoveContainer" containerID="699c780335e11acca2d2d73a73e7614814dc48cf5898632c4e21e78c178c6180" Nov 03 18:04:15.536854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.536824 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" event={"ID":"fe87c6e6-03da-45a0-9cae-2854d65e15ef","Type":"ContainerDied","Data":"157c703a7f6d24b8bc4c1d710087892e93af1ff6959affc09bcc048ba7e37a9d"} Nov 03 18:04:15.537030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.536942 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8" Nov 03 18:04:15.548675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.548651 4980 scope.go:122] "RemoveContainer" containerID="cd29504e96a153a0aab7ca77acfd1946d063aecf5e3957bce93188ba0e5c30ad" Nov 03 18:04:15.730708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.730673 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:04:15.734858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.734834 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-wpjv8"] Nov 03 18:04:15.846697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.846611 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:04:15.850629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:15.850600 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-394741e0-openshift-default-545f5658c5-w9ljz"] Nov 03 18:04:16.299277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.299240 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66d23304-b75e-47d1-b37d-60a707459bac" path="/var/lib/kubelet/pods/66d23304-b75e-47d1-b37d-60a707459bac/volumes" Nov 03 18:04:16.300029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.300008 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" path="/var/lib/kubelet/pods/b0390d5c-53b5-4f5f-8b12-308fce3b5c1d/volumes" Nov 03 18:04:16.300549 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.300527 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" path="/var/lib/kubelet/pods/d0d47cb0-cb62-4ae4-aa0a-348421ca18d6/volumes" Nov 03 18:04:16.301599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.301559 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" path="/var/lib/kubelet/pods/fe87c6e6-03da-45a0-9cae-2854d65e15ef/volumes" Nov 03 18:04:16.810813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.810777 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:16.810994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:16.810832 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:19.605601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:19.598752 4980 patch_prober.go:29] interesting pod/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:19.605601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:19.598817 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:20.427747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.427717 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 18:04:20.439495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.439464 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 18:04:20.464670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464635 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.464816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464717 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.464816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464760 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.464951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464845 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.464951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464904 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.464951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464942 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.464990 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.465105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465024 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.465105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465074 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465019 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.465242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465136 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.465282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465237 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465263 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465284 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.465346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465300 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.465437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465353 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465402 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465454 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.465597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465547 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465674 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465655 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\") pod \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\" (UID: \"e948d145-ce64-4ecc-b176-8f2c5b65d9e6\") " Nov 03 18:04:20.465752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.465735 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\") pod \"0e6d0a17-3921-48ef-8120-8b66793229a7\" (UID: \"0e6d0a17-3921-48ef-8120-8b66793229a7\") " Nov 03 18:04:20.467136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.466651 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.467136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.466675 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.467136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.466689 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.467136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.466913 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:20.467391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.467362 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:20.467819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.467771 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.468920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.468891 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.469107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.469082 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.469284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.469260 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.473777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.473748 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.474973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.474946 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:20.477606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.477554 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "kube-api-access-8xvjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.479376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.479341 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.479481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.479457 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:20.479481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.479465 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.479980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.479942 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy" pod "0e6d0a17-3921-48ef-8120-8b66793229a7" (UID: "0e6d0a17-3921-48ef-8120-8b66793229a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.480324 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.480301 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "kube-api-access-cswc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.483244 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.483216 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token" pod "e948d145-ce64-4ecc-b176-8f2c5b65d9e6" (UID: "e948d145-ce64-4ecc-b176-8f2c5b65d9e6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.560954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.560926 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 18:04:20.564848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.564826 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 18:04:20.568299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568270 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568306 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568323 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568339 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8xvjj\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-kube-api-access-8xvjj\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568355 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568369 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568382 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568396 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e6d0a17-3921-48ef-8120-8b66793229a7-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568410 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568423 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e6d0a17-3921-48ef-8120-8b66793229a7-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568437 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568453 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568466 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e6d0a17-3921-48ef-8120-8b66793229a7-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568479 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.568624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.568494 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cswc8\" (UniqueName: \"kubernetes.io/projected/e948d145-ce64-4ecc-b176-8f2c5b65d9e6-kube-api-access-cswc8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.669773 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669736 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669798 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669845 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669909 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669935 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.669982 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670041 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670048 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670092 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670121 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670169 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670202 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670229 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.670264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670234 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670284 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670341 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670404 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\") pod \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\" (UID: \"b873742f-df2f-4e9d-8432-be1a7b4b3bbf\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670449 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670504 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670517 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.671173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.670622 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\") pod \"2b85d0fd-a322-40fb-9cec-0da989e071ab\" (UID: \"2b85d0fd-a322-40fb-9cec-0da989e071ab\") " Nov 03 18:04:20.671582 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.671381 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.671763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.671737 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:20.671887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.671860 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672016 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2b85d0fd-a322-40fb-9cec-0da989e071ab-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672019 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672040 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672056 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672068 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.672093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672077 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.672553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672105 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.672553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672081 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.672553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672133 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.672553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672143 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.673021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.672983 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:20.673368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.673349 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.673888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.673851 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.674155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.674126 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.674451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.674430 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp" pod "b873742f-df2f-4e9d-8432-be1a7b4b3bbf" (UID: "b873742f-df2f-4e9d-8432-be1a7b4b3bbf"). InnerVolumeSpecName "kube-api-access-b9gvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.674620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.674560 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "kube-api-access-d5fxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:20.674711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.674643 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:20.675154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.675127 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy" pod "2b85d0fd-a322-40fb-9cec-0da989e071ab" (UID: "2b85d0fd-a322-40fb-9cec-0da989e071ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:20.702612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.702585 4980 generic.go:356] "Generic (PLEG): container finished" podID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerID="c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4" exitCode=0 Nov 03 18:04:20.702788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.702627 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" Nov 03 18:04:20.702788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.702674 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" event={"ID":"2b85d0fd-a322-40fb-9cec-0da989e071ab","Type":"ContainerDied","Data":"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4"} Nov 03 18:04:20.702788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.702729 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn" event={"ID":"2b85d0fd-a322-40fb-9cec-0da989e071ab","Type":"ContainerDied","Data":"dc7d4d4ca9c17204b80f1492f5b09ada12d291205db8109b21de8fc09022bddb"} Nov 03 18:04:20.702788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.702757 4980 scope.go:122] "RemoveContainer" containerID="c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4" Nov 03 18:04:20.704699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.704673 4980 generic.go:356] "Generic (PLEG): container finished" podID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerID="826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69" exitCode=0 Nov 03 18:04:20.704811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.704727 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" event={"ID":"e948d145-ce64-4ecc-b176-8f2c5b65d9e6","Type":"ContainerDied","Data":"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69"} Nov 03 18:04:20.704811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.704783 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" event={"ID":"e948d145-ce64-4ecc-b176-8f2c5b65d9e6","Type":"ContainerDied","Data":"462effef535a88530959620dcc5fb8a7917dcccf5ab825fcb1f45514b0813458"} Nov 03 18:04:20.704811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.704743 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw" Nov 03 18:04:20.706522 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.706499 4980 generic.go:356] "Generic (PLEG): container finished" podID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerID="c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3" exitCode=0 Nov 03 18:04:20.706703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.706684 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" Nov 03 18:04:20.706821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.706693 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" event={"ID":"b873742f-df2f-4e9d-8432-be1a7b4b3bbf","Type":"ContainerDied","Data":"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3"} Nov 03 18:04:20.706886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.706852 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph" event={"ID":"b873742f-df2f-4e9d-8432-be1a7b4b3bbf","Type":"ContainerDied","Data":"c1e5aa17c89f8dbc3dfe57a9f2168c5144ad9454fc694270aca749f291262a99"} Nov 03 18:04:20.712860 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.712839 4980 generic.go:356] "Generic (PLEG): container finished" podID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerID="1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54" exitCode=0 Nov 03 18:04:20.712962 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.712870 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" event={"ID":"0e6d0a17-3921-48ef-8120-8b66793229a7","Type":"ContainerDied","Data":"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54"} Nov 03 18:04:20.712962 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.712909 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" event={"ID":"0e6d0a17-3921-48ef-8120-8b66793229a7","Type":"ContainerDied","Data":"5864f42cbaa1c927325b87ad14ce0d77b487540ec2082920899facaf19e568df"} Nov 03 18:04:20.713089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.712993 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf" Nov 03 18:04:20.717115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.717074 4980 scope.go:122] "RemoveContainer" containerID="c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4" Nov 03 18:04:20.717635 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:20.717380 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4\": container with ID starting with c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4 not found: ID does not exist" containerID="c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4" Nov 03 18:04:20.717635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.717415 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4"} err="failed to get container status \"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4\": rpc error: code = NotFound desc = could not find container \"c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4\": container with ID starting with c750808d152db5317a7cc2f8352c9c274baeb75ea8c371137e1f4fafedf342b4 not found: ID does not exist" Nov 03 18:04:20.717635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.717440 4980 scope.go:122] "RemoveContainer" containerID="826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69" Nov 03 18:04:20.730801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.730776 4980 scope.go:122] "RemoveContainer" containerID="826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69" Nov 03 18:04:20.731232 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:20.731210 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69\": container with ID starting with 826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69 not found: ID does not exist" containerID="826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69" Nov 03 18:04:20.731327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.731257 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69"} err="failed to get container status \"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69\": rpc error: code = NotFound desc = could not find container \"826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69\": container with ID starting with 826ea9abde4d4035be90cb86ddaf13217116a181265f26bffd8124faa81f3a69 not found: ID does not exist" Nov 03 18:04:20.731327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.731279 4980 scope.go:122] "RemoveContainer" containerID="c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3" Nov 03 18:04:20.745979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.745894 4980 scope.go:122] "RemoveContainer" containerID="c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3" Nov 03 18:04:20.746460 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:20.746435 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3\": container with ID starting with c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3 not found: ID does not exist" containerID="c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3" Nov 03 18:04:20.746611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.746466 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3"} err="failed to get container status \"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3\": rpc error: code = NotFound desc = could not find container \"c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3\": container with ID starting with c863ea6b6043febc3ca85ca0d1dabdf70cf75602e225b2cb30db529e0fae62c3 not found: ID does not exist" Nov 03 18:04:20.746611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.746484 4980 scope.go:122] "RemoveContainer" containerID="1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54" Nov 03 18:04:20.758080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.758059 4980 scope.go:122] "RemoveContainer" containerID="1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54" Nov 03 18:04:20.758422 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:20.758395 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54\": container with ID starting with 1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54 not found: ID does not exist" containerID="1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54" Nov 03 18:04:20.758504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.758429 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54"} err="failed to get container status \"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54\": rpc error: code = NotFound desc = could not find container \"1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54\": container with ID starting with 1349a4dc54cfa62ca05ac48764b075533dd5d60f301eb40cb7674e8658f6ea54 not found: ID does not exist" Nov 03 18:04:20.773515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773481 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773519 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773535 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773548 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2b85d0fd-a322-40fb-9cec-0da989e071ab-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773608 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773619 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773629 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773638 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5fxn\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-kube-api-access-d5fxn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773653 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9gvp\" (UniqueName: \"kubernetes.io/projected/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-kube-api-access-b9gvp\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773667 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773682 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2b85d0fd-a322-40fb-9cec-0da989e071ab-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:20.773980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:20.773697 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b873742f-df2f-4e9d-8432-be1a7b4b3bbf-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:21.120691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.120655 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 18:04:21.128425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.128386 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-znjtn"] Nov 03 18:04:21.301648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.290391 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 18:04:21.383638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.383072 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-mxbgf"] Nov 03 18:04:21.465667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.465581 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 18:04:21.490828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.478768 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-728ph"] Nov 03 18:04:21.555823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.555687 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 18:04:21.606596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:21.594889 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-548ca1-openshift-default-698f7745bc-bfkvw"] Nov 03 18:04:22.302525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:22.302487 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" path="/var/lib/kubelet/pods/0e6d0a17-3921-48ef-8120-8b66793229a7/volumes" Nov 03 18:04:22.304026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:22.304000 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" path="/var/lib/kubelet/pods/2b85d0fd-a322-40fb-9cec-0da989e071ab/volumes" Nov 03 18:04:22.304915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:22.304894 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" path="/var/lib/kubelet/pods/b873742f-df2f-4e9d-8432-be1a7b4b3bbf/volumes" Nov 03 18:04:22.306204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:22.306183 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" path="/var/lib/kubelet/pods/e948d145-ce64-4ecc-b176-8f2c5b65d9e6/volumes" Nov 03 18:04:23.965217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:23.965169 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.318221 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319287 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319309 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319343 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319351 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319364 4980 cpu_manager.go:414] "containerMap: removing container" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319371 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319383 4980 cpu_manager.go:414] "containerMap: removing container" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319391 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319402 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f064fc2c-b8af-4f89-82ec-c7ac36728720" containerName="payload-processing" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319410 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f064fc2c-b8af-4f89-82ec-c7ac36728720" containerName="payload-processing" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319430 4980 cpu_manager.go:414] "containerMap: removing container" podUID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" containerName="payload-pre-processing" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319437 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" containerName="payload-pre-processing" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319451 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319458 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319487 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319494 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319509 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319515 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319531 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319537 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319550 4980 cpu_manager.go:414] "containerMap: removing container" podUID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" containerName="payload-processing" Nov 03 18:04:24.374649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.319557 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" containerName="payload-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389848 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389873 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389898 4980 cpu_manager.go:414] "containerMap: removing container" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389905 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389931 4980 cpu_manager.go:414] "containerMap: removing container" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389937 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389944 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" containerName="payload-pre-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.389951 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" containerName="payload-pre-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390173 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="55efd5dc-3fa5-448e-87e3-6510d7e039a2" containerName="payload-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390191 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e6d0a17-3921-48ef-8120-8b66793229a7" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390206 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="66d23304-b75e-47d1-b37d-60a707459bac" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390215 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b85d0fd-a322-40fb-9cec-0da989e071ab" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390224 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3fa0a0b-f2aa-448d-b433-5e01e68e9a1f" containerName="payload-pre-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390235 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390247 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b873742f-df2f-4e9d-8432-be1a7b4b3bbf" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390256 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e948d145-ce64-4ecc-b176-8f2c5b65d9e6" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390266 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390274 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f064fc2c-b8af-4f89-82ec-c7ac36728720" containerName="payload-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390285 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac5228e2-f3c8-453c-98a0-e136b4b13cf6" containerName="payload-pre-processing" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390295 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5829e122-cc6e-4a48-9468-868a1eefdf37" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390303 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390317 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c2d53fa-70fe-4707-a617-6b6ccc87321c" containerName="istio-proxy" Nov 03 18:04:24.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.390328 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b0390d5c-53b5-4f5f-8b12-308fce3b5c1d" containerName="istio-proxy" Nov 03 18:04:24.496030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.439957 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/health\": context deadline exceeded" Nov 03 18:04:24.496030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.447995 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:04:24.496030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.452023 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.593944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.559537 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-62aead-openshift-default-dockercfg-g55v6\"" Nov 03 18:04:24.711744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.646274 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.711744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.646362 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.711744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.646396 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.711744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.646454 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.711744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.646555 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqzqt\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.861604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.850650 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.861604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.850713 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.861604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.850755 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.872608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.789039 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:24.872608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.792318 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:24.872608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.869455 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:24.872608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.871446 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:04:24.985651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.984923 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:25.001856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.986284 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:04:25.001856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:24.994432 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.002852 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003411 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003444 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003493 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003553 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqzqt\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003850 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.006642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.003321 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.013635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.012756 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.013635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.012815 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.013635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.012854 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.013635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.013100 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.018606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.013966 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.082073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.030804 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.082073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.031326 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.082073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.046910 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.089638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.083558 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.110273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.110239 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.156599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.143775 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqzqt\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\") pod \"e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.162595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.158029 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:25.170595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.163881 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:25.171225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.171190 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:25.458597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:25.446719 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:26.053130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.048480 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:26.053130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.048904 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:26.053130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.049197 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": context deadline exceeded" Nov 03 18:04:26.111928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.070129 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:04:26.111928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.070200 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:04:26.111928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.070241 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:04:26.111928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.072549 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:26.135415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.113715 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:04:26.195667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.153947 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:04:26.312690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.312616 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:04:26.343620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.343563 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:26.444216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.444185 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:04:26.444477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.444465 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:04:26.444678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.444664 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:04:26.444935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.444912 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:04:26.499976 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.499945 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:04:26.523897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.523859 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:26.524131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.524102 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:26.524234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:26.524162 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:27.125001 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.124971 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:04:27.196189 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.196148 4980 cpu_manager.go:414] "containerMap: removing container" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:27.215729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.215681 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d47cb0-cb62-4ae4-aa0a-348421ca18d6" containerName="istio-proxy" Nov 03 18:04:27.285050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.285009 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe87c6e6-03da-45a0-9cae-2854d65e15ef" containerName="istio-proxy" Nov 03 18:04:27.503204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.503154 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.522335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.522291 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn5pd\" (UniqueName: \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.522878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.522848 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.552360 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.552325 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerStarted","Data":"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e"} Nov 03 18:04:27.553921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.553893 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerStarted","Data":"2cd78c0dad6b96d4666ac1d17fada7aedd08088e00575f405a7f2ec54da0a57d"} Nov 03 18:04:27.566600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.562441 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d9a300\"" Nov 03 18:04:27.567776 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.567752 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d9a300-dockercfg-dg6tq\"" Nov 03 18:04:27.567989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.566864 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:04:27.604453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.600489 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:04:27.619651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.619623 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.621074 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.621050 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:04:27.624424 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.624399 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.625066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.624758 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn5pd\" (UniqueName: \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.626833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.626625 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9h2h\" (UniqueName: \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.628096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.628073 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.629199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.629175 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.693846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.692355 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn5pd\" (UniqueName: \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\") pod \"payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.736904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.734136 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9h2h\" (UniqueName: \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.736904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.734332 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.736904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.735200 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.742598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.742148 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podStartSLOduration=3.742136596 podStartE2EDuration="3.742136596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:27.741646768 +0000 UTC m=+1847.570730372" watchObservedRunningTime="2026-11-03 18:04:27.742136596 +0000 UTC m=+1847.571220198" Nov 03 18:04:27.798077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.798007 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9h2h\" (UniqueName: \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\") pod \"payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:27.917921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.917876 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:27.957960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:27.945556 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:28.181744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.181645 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:28.189098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.188751 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:28.417628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.417549 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 18:04:28.417785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.417762 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" containerID="cri-o://74b5e3886e83ca52e6b5e2c119071f2755b86a5ee59b13dcb29ece4d6efaf9f8" gracePeriod=30 Nov 03 18:04:28.490811 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:04:28.490773 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8378a5_4ac6_4726_81ae_070a3937d984.slice/crio-528cb9eb6d9b88cb61fe1420cd9385e5a7fb12df0e1832f2255324eb4751b89c WatchSource:0}: Error finding container 528cb9eb6d9b88cb61fe1420cd9385e5a7fb12df0e1832f2255324eb4751b89c: Status 404 returned error can't find the container with id 528cb9eb6d9b88cb61fe1420cd9385e5a7fb12df0e1832f2255324eb4751b89c Nov 03 18:04:28.492636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.492451 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:04:28.527685 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.527614 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:04:28.538026 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:04:28.537995 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6432b682_dbb3_493b_b52f_374822e176c7.slice/crio-a885dc74fa852f8c6164bcb690c7ef933e3e9d03be389257fae9f7187511186b WatchSource:0}: Error finding container a885dc74fa852f8c6164bcb690c7ef933e3e9d03be389257fae9f7187511186b: Status 404 returned error can't find the container with id a885dc74fa852f8c6164bcb690c7ef933e3e9d03be389257fae9f7187511186b Nov 03 18:04:28.612056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.612024 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" event={"ID":"3e8378a5-4ac6-4726-81ae-070a3937d984","Type":"ContainerStarted","Data":"528cb9eb6d9b88cb61fe1420cd9385e5a7fb12df0e1832f2255324eb4751b89c"} Nov 03 18:04:28.613905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.613880 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" event={"ID":"6432b682-dbb3-493b-b52f-374822e176c7","Type":"ContainerStarted","Data":"a885dc74fa852f8c6164bcb690c7ef933e3e9d03be389257fae9f7187511186b"} Nov 03 18:04:28.614612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.614554 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:28.615714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:28.615692 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:04:29.622633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.622596 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" event={"ID":"3e8378a5-4ac6-4726-81ae-070a3937d984","Type":"ContainerStarted","Data":"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e"} Nov 03 18:04:29.623086 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.622703 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:29.624914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.624879 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" event={"ID":"6432b682-dbb3-493b-b52f-374822e176c7","Type":"ContainerStarted","Data":"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7"} Nov 03 18:04:29.625054 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.624958 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:29.747323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.747260 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" podStartSLOduration=2.747246079 podStartE2EDuration="2.747246079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:29.742271461 +0000 UTC m=+1849.571355077" watchObservedRunningTime="2026-11-03 18:04:29.747246079 +0000 UTC m=+1849.576329681" Nov 03 18:04:29.964375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:29.964307 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" podStartSLOduration=3.964284181 podStartE2EDuration="3.964284181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:29.896874053 +0000 UTC m=+1849.725957656" watchObservedRunningTime="2026-11-03 18:04:29.964284181 +0000 UTC m=+1849.793367787" Nov 03 18:04:33.278774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.278731 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:04:33.279284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.278811 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:04:33.505605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.505545 4980 patch_prober.go:29] interesting pod/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" start-of-body= Nov 03 18:04:33.505765 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.505632 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.81:15021/healthz/ready\": dial tcp 10.132.0.81:15021: connect: connection refused" Nov 03 18:04:33.681868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.681838 4980 generic.go:356] "Generic (PLEG): container finished" podID="27c69b51-d406-418a-9716-b418ac458773" containerID="74b5e3886e83ca52e6b5e2c119071f2755b86a5ee59b13dcb29ece4d6efaf9f8" exitCode=0 Nov 03 18:04:33.682047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.681908 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" event={"ID":"27c69b51-d406-418a-9716-b418ac458773","Type":"ContainerDied","Data":"74b5e3886e83ca52e6b5e2c119071f2755b86a5ee59b13dcb29ece4d6efaf9f8"} Nov 03 18:04:33.778031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.777999 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 18:04:33.900487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900388 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.900694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900550 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.900777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900694 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.900839 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900778 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.900937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900892 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:33.901000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900942 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:04:33.901000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.900981 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.901115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.901035 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.901115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.901108 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.901256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.901209 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:33.901833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.901801 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.901963 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.901893 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\") pod \"27c69b51-d406-418a-9716-b418ac458773\" (UID: \"27c69b51-d406-418a-9716-b418ac458773\") " Nov 03 18:04:33.902237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.902163 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.902339 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.904188 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.904224 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.904242 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.904264 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/27c69b51-d406-418a-9716-b418ac458773-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:33.905238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.904279 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:33.907432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.907376 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:04:33.917759 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.917716 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:04:33.917977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.917733 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:33.918556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:33.918528 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf" pod "27c69b51-d406-418a-9716-b418ac458773" (UID: "27c69b51-d406-418a-9716-b418ac458773"). InnerVolumeSpecName "kube-api-access-8swdf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:04:34.011337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.011238 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/27c69b51-d406-418a-9716-b418ac458773-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:34.011337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.011280 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8swdf\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-kube-api-access-8swdf\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:34.011337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.011298 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/27c69b51-d406-418a-9716-b418ac458773-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:34.011337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.011312 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/27c69b51-d406-418a-9716-b418ac458773-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:04:34.924954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.924912 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" event={"ID":"27c69b51-d406-418a-9716-b418ac458773","Type":"ContainerDied","Data":"636bdf0118078e9b9d03c6a1e234e1ed6cf49bc37e125ffd66ebd15afbc239c2"} Nov 03 18:04:34.925819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.925788 4980 scope.go:122] "RemoveContainer" containerID="74b5e3886e83ca52e6b5e2c119071f2755b86a5ee59b13dcb29ece4d6efaf9f8" Nov 03 18:04:34.926112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:34.925697 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c" Nov 03 18:04:35.153500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.153461 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 18:04:35.203734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.203704 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-8rc4c"] Nov 03 18:04:35.935121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.935086 4980 generic.go:356] "Generic (PLEG): container finished" podID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerID="6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297" exitCode=0 Nov 03 18:04:35.935648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.935165 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerDied","Data":"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297"} Nov 03 18:04:35.935785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.935766 4980 scope.go:122] "RemoveContainer" containerID="6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297" Nov 03 18:04:35.937481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.937451 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:35.937645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.937538 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:35.937726 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:35.937639 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:36.332715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:36.331399 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="27c69b51-d406-418a-9716-b418ac458773" path="/var/lib/kubelet/pods/27c69b51-d406-418a-9716-b418ac458773/volumes" Nov 03 18:04:36.951870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:36.951839 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerStarted","Data":"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56"} Nov 03 18:04:37.576604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:37.567910 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:04:37.576604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:37.568041 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:37.576604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:37.568085 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:38.569151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:38.568497 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:38.569151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:38.568622 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:39.577599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:39.572970 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:39.577599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:39.573046 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:40.573599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:40.570613 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:40.573599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:40.570695 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:40.634494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:40.634464 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:04:40.634955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:40.634540 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:04:41.571652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:41.568933 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:41.571652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:41.569010 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:42.580597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:42.576963 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" start-of-body= Nov 03 18:04:42.580597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:42.577040 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.111:15021/healthz/ready\": dial tcp 10.132.0.111:15021: connect: connection refused" Nov 03 18:04:43.572503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:43.572473 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:04:43.572994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:43.572970 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:04:43.573630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:43.573612 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:04:44.520502 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:04:44.520465 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:04:47.542301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.542260 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:04:47.543409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.543375 4980 cpu_manager.go:414] "containerMap: removing container" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" Nov 03 18:04:47.543641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.543619 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" Nov 03 18:04:47.543999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.543980 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="27c69b51-d406-418a-9716-b418ac458773" containerName="istio-proxy" Nov 03 18:04:47.570596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.560098 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.669601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.668139 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-8dfc64-openshift-default-dockercfg-xd4hn\"" Nov 03 18:04:47.755253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.755200 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.755452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.755325 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcvp2\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.767608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.755496 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.767608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.755719 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.767608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.755786 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.767608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.756051 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.767608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.756088 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.859132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.756286 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.859132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.850453 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:47.981602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.972637 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998079 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998194 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998279 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998394 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998486 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcvp2\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998679 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998741 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998806 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998889 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.998920 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:47.999523 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.004545 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.004896 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.006652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.005544 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.034438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.028478 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.034438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.028726 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.072757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.072728 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.073066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.073042 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcvp2\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\") pod \"e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.150027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.149949 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:48.599749 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.599718 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:04:48.600218 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:04:48.600193 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f8a1b88_a32e_400e_9d3f_026254e95f4a.slice/crio-583b61df9c4bb0145034140c7a0a665d15aefe123b09f519bcbed7e33c03c832 WatchSource:0}: Error finding container 583b61df9c4bb0145034140c7a0a665d15aefe123b09f519bcbed7e33c03c832: Status 404 returned error can't find the container with id 583b61df9c4bb0145034140c7a0a665d15aefe123b09f519bcbed7e33c03c832 Nov 03 18:04:48.603186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.603151 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:48.603282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.603219 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:48.603282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:48.603253 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:04:49.071505 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:49.071466 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerStarted","Data":"7da59dedf7ce897534bc5d6fce1d53f480b52cf0c4299ad6b848d5a6282af055"} Nov 03 18:04:49.071740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:49.071523 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerStarted","Data":"583b61df9c4bb0145034140c7a0a665d15aefe123b09f519bcbed7e33c03c832"} Nov 03 18:04:49.150915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:49.150829 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:49.157415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:49.157389 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:49.203412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:49.203360 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podStartSLOduration=2.203345544 podStartE2EDuration="2.203345544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:49.201239444 +0000 UTC m=+1869.030323047" watchObservedRunningTime="2026-11-03 18:04:49.203345544 +0000 UTC m=+1869.032429146" Nov 03 18:04:50.081635 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:50.081602 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:50.083251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:50.083223 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:04:57.362331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.362299 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 18:04:57.436625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.425615 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 18:04:57.451597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.440824 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" containerID="cri-o://784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4" gracePeriod=30 Nov 03 18:04:57.471599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.468627 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 18:04:57.471599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.468911 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" containerID="cri-o://d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904" gracePeriod=30 Nov 03 18:04:57.509595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.508664 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 18:04:57.509595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:57.508955 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" containerID="cri-o://50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b" gracePeriod=30 Nov 03 18:04:58.190212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.188997 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:04:58.194096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.194065 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 18:04:58.194500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.194467 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" containerID="cri-o://225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76" gracePeriod=30 Nov 03 18:04:58.194890 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.194866 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.244442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.235406 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 18:04:58.244442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.235720 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" containerID="cri-o://ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb" gracePeriod=30 Nov 03 18:04:58.244442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.235841 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 18:04:58.244442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.236046 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" containerID="cri-o://51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c" gracePeriod=30 Nov 03 18:04:58.267545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.267501 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 18:04:58.267772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.267611 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:04:58.267845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.267802 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" containerID="cri-o://9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0" gracePeriod=30 Nov 03 18:04:58.273266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.273238 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-8dfc64\"" Nov 03 18:04:58.274649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.274626 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-8dfc64-dockercfg-fjws9\"" Nov 03 18:04:58.285838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.285778 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" containerID="cri-o://ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f" gracePeriod=30 Nov 03 18:04:58.338713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.338681 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcxpl\" (UniqueName: \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.338912 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.338889 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.439979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.439935 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.440437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.440149 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcxpl\" (UniqueName: \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.440824 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.440803 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.504153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.504120 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcxpl\" (UniqueName: \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\") pod \"payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.570472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.570444 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:58.961983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.961946 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:04:58.962174 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:04:58.962151 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea5267ea_f827_4810_a5bb_93032a869f8e.slice/crio-23fa35c8c562c4d65da707bf63d99ed3de3f95dd77d408804c8f888905b69c9a WatchSource:0}: Error finding container 23fa35c8c562c4d65da707bf63d99ed3de3f95dd77d408804c8f888905b69c9a: Status 404 returned error can't find the container with id 23fa35c8c562c4d65da707bf63d99ed3de3f95dd77d408804c8f888905b69c9a Nov 03 18:04:58.992209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.992175 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:58.992337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:58.992237 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:59.294959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.294878 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" event={"ID":"ea5267ea-f827-4810-a5bb-93032a869f8e","Type":"ContainerStarted","Data":"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821"} Nov 03 18:04:59.294959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.294929 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" event={"ID":"ea5267ea-f827-4810-a5bb-93032a869f8e","Type":"ContainerStarted","Data":"23fa35c8c562c4d65da707bf63d99ed3de3f95dd77d408804c8f888905b69c9a"} Nov 03 18:04:59.295145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.295022 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:04:59.321357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.321324 4980 patch_prober.go:29] interesting pod/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:04:59.321531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.321382 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:04:59.425599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:04:59.425261 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" podStartSLOduration=1.425241594 podStartE2EDuration="1.425241594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:04:59.4176042 +0000 UTC m=+1879.246687803" watchObservedRunningTime="2026-11-03 18:04:59.425241594 +0000 UTC m=+1879.254325199" Nov 03 18:05:01.414716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:01.414680 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:05:01.415123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:01.414755 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:05:02.844191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.844158 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 18:05:02.885173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.885144 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 18:05:02.917811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.917789 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 18:05:02.925013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.924990 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925023 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925088 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.925254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925159 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925196 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.925254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925245 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.925412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925274 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.925412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925310 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925354 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925384 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.925637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925437 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.925637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925432 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.925637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925446 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.925637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925511 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.925637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925597 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925661 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925711 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925747 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925787 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925831 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.926055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.925939 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.926603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.926534 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.926928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.926678 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\") pod \"e692a971-c058-4d22-8bdd-0992fef3c254\" (UID: \"e692a971-c058-4d22-8bdd-0992fef3c254\") " Nov 03 18:05:02.926928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.926698 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:02.926928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.926765 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\") pod \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\" (UID: \"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee\") " Nov 03 18:05:02.927234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.926951 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:02.927234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927191 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.927344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927330 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.927753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927733 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.927831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927760 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.927831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927776 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.927831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927791 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e692a971-c058-4d22-8bdd-0992fef3c254-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.927831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927805 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.927831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927819 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.928081 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927832 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.928081 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927846 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.928081 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927860 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.928081 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.927873 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:02.928596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.928547 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.928703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.928681 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:02.928935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.928916 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:02.929019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.928951 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:02.929116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.929098 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "kube-api-access-8d24v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:02.929415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.929400 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:02.929504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.929486 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf" pod "e692a971-c058-4d22-8bdd-0992fef3c254" (UID: "e692a971-c058-4d22-8bdd-0992fef3c254"). InnerVolumeSpecName "kube-api-access-7sztf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:02.929997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:02.929980 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo" pod "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" (UID: "7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.028970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.028931 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.028990 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029044 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029129 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029162 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029226 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029221 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.029498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029354 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.029498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029463 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.029637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029588 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.030315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029803 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.030315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.029844 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\") pod \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\" (UID: \"f03b91bd-ab96-4797-ad4c-e214ba415a4a\") " Nov 03 18:05:03.030315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.030188 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.030315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.030282 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.031388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031360 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031395 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031411 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031411 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031426 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e692a971-c058-4d22-8bdd-0992fef3c254-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031440 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031454 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031466 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031478 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031491 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d24v\" (UniqueName: \"kubernetes.io/projected/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee-kube-api-access-8d24v\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031503 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7sztf\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-kube-api-access-7sztf\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031513 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031521 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e692a971-c058-4d22-8bdd-0992fef3c254-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.031593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031535 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e692a971-c058-4d22-8bdd-0992fef3c254-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.032161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031857 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "kube-api-access-95mbz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.032161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031908 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.032161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.031913 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo" pod "f03b91bd-ab96-4797-ad4c-e214ba415a4a" (UID: "f03b91bd-ab96-4797-ad4c-e214ba415a4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.140772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.140675 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.140772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.140707 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.140772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.140721 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f03b91bd-ab96-4797-ad4c-e214ba415a4a-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.140772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.140735 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95mbz\" (UniqueName: \"kubernetes.io/projected/f03b91bd-ab96-4797-ad4c-e214ba415a4a-kube-api-access-95mbz\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.322737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.322704 4980 generic.go:356] "Generic (PLEG): container finished" podID="e692a971-c058-4d22-8bdd-0992fef3c254" containerID="d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904" exitCode=0 Nov 03 18:05:03.322889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.322776 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" Nov 03 18:05:03.322889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.322850 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" event={"ID":"e692a971-c058-4d22-8bdd-0992fef3c254","Type":"ContainerDied","Data":"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904"} Nov 03 18:05:03.323018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.322900 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb" event={"ID":"e692a971-c058-4d22-8bdd-0992fef3c254","Type":"ContainerDied","Data":"b0d170c39acac96b6ad00daaee91bef950406a123bffdbaf36637f2e4c7ef5a7"} Nov 03 18:05:03.323018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.322929 4980 scope.go:122] "RemoveContainer" containerID="d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904" Nov 03 18:05:03.324780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.324761 4980 generic.go:356] "Generic (PLEG): container finished" podID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerID="50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b" exitCode=0 Nov 03 18:05:03.324910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.324897 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" event={"ID":"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee","Type":"ContainerDied","Data":"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b"} Nov 03 18:05:03.324973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.324929 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" event={"ID":"7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee","Type":"ContainerDied","Data":"7c4ed8a0c38782834fb6051c7285e607818d1d5c50c3dcde940cd63e0e6ea363"} Nov 03 18:05:03.325029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.324989 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt" Nov 03 18:05:03.329666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.329639 4980 generic.go:356] "Generic (PLEG): container finished" podID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerID="784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4" exitCode=0 Nov 03 18:05:03.329815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.329681 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" event={"ID":"f03b91bd-ab96-4797-ad4c-e214ba415a4a","Type":"ContainerDied","Data":"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4"} Nov 03 18:05:03.329815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.329709 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" Nov 03 18:05:03.329815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.329718 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq" event={"ID":"f03b91bd-ab96-4797-ad4c-e214ba415a4a","Type":"ContainerDied","Data":"ba7e2df6e227e12e6a392dba96f7d7c0c5ce9bb23b106997ccf142137688bbc6"} Nov 03 18:05:03.512479 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.512452 4980 scope.go:122] "RemoveContainer" containerID="d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904" Nov 03 18:05:03.513116 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:03.513089 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904\": container with ID starting with d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904 not found: ID does not exist" containerID="d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904" Nov 03 18:05:03.513187 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.513132 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904"} err="failed to get container status \"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904\": rpc error: code = NotFound desc = could not find container \"d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904\": container with ID starting with d7a34d84848981fd69c97fec9002a4c5bc25a0a8bf64485a6c8a7d2e08d86904 not found: ID does not exist" Nov 03 18:05:03.513187 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.513156 4980 scope.go:122] "RemoveContainer" containerID="50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b" Nov 03 18:05:03.532589 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.532270 4980 scope.go:122] "RemoveContainer" containerID="50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b" Nov 03 18:05:03.533154 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:03.532803 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b\": container with ID starting with 50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b not found: ID does not exist" containerID="50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b" Nov 03 18:05:03.533154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.532844 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b"} err="failed to get container status \"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b\": rpc error: code = NotFound desc = could not find container \"50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b\": container with ID starting with 50591db38b5600caff9bc1bf1d050927536efda9e2e01393f21225a1cec2b49b not found: ID does not exist" Nov 03 18:05:03.533154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.532868 4980 scope.go:122] "RemoveContainer" containerID="784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4" Nov 03 18:05:03.559676 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.559651 4980 scope.go:122] "RemoveContainer" containerID="784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4" Nov 03 18:05:03.560030 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:03.560006 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4\": container with ID starting with 784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4 not found: ID does not exist" containerID="784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4" Nov 03 18:05:03.560124 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.560040 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4"} err="failed to get container status \"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4\": rpc error: code = NotFound desc = could not find container \"784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4\": container with ID starting with 784a790d3b25da70d9d05e8736dfe4afd1d952e3412f6992c621274a686fccc4 not found: ID does not exist" Nov 03 18:05:03.626446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.626386 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 18:05:03.627218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.627196 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 18:05:03.632738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.632711 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-54kbq"] Nov 03 18:05:03.683164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.683140 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 18:05:03.746328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.746296 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 18:05:03.751194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.751167 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-m64fb"] Nov 03 18:05:03.753234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753213 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753246 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753304 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753330 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753553 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.753727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753561 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.753795 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753728 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.753795 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753739 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.753892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753790 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.753892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753807 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753847 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.753892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.753874 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.754117 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754095 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.754236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754196 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.754236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754223 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754260 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754297 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754329 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754410 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754448 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754481 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754524 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\") pod \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\" (UID: \"b4a64170-0c8b-4f86-ba3a-e84320fa0472\") " Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754525 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754610 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\") pod \"a01e1041-e6a8-46a1-99bf-8f5f622df595\" (UID: \"a01e1041-e6a8-46a1-99bf-8f5f622df595\") " Nov 03 18:05:03.754703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754631 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.755006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.754986 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.755446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755421 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755454 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755545 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755595 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755666 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755695 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755708 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.755764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755722 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.756134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755936 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.756134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.755942 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.756306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.756282 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.756796 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.756769 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.757432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.757411 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.757842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.757820 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.757998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.757907 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.758347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.758326 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng" pod "b4a64170-0c8b-4f86-ba3a-e84320fa0472" (UID: "b4a64170-0c8b-4f86-ba3a-e84320fa0472"). InnerVolumeSpecName "kube-api-access-qkxng". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.758817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.758799 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 18:05:03.759033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.759015 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "kube-api-access-lcsvs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.759384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.759363 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy" pod "a01e1041-e6a8-46a1-99bf-8f5f622df595" (UID: "a01e1041-e6a8-46a1-99bf-8f5f622df595"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.763327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.763306 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 18:05:03.767174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.767151 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 18:05:03.857172 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857137 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857192 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857214 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857259 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857275 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857310 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857329 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857380 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857442 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857506 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857508 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857519 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857545 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857601 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857630 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857688 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.857739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857716 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857766 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857815 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857841 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857876 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857882 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857916 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.857956 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858056 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858061 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858095 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858136 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858189 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858253 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\") pod \"6df1d58a-ceab-439e-8fa7-7496a990fa97\" (UID: \"6df1d58a-ceab-439e-8fa7-7496a990fa97\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858288 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\") pod \"4117524c-c6d8-46d1-8516-9138033f2b06\" (UID: \"4117524c-c6d8-46d1-8516-9138033f2b06\") " Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858338 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858685 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858744 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.858909 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859100 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859412 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859509 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.859721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859627 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859866 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\") pod \"f828a59b-451c-4fea-a029-c8a3af694295\" (UID: \"f828a59b-451c-4fea-a029-c8a3af694295\") " Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.859974 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.860977 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861014 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861055 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861071 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861086 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f828a59b-451c-4fea-a029-c8a3af694295-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861100 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6df1d58a-ceab-439e-8fa7-7496a990fa97-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861115 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861129 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861144 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861160 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861174 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4117524c-c6d8-46d1-8516-9138033f2b06-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861189 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lcsvs\" (UniqueName: \"kubernetes.io/projected/a01e1041-e6a8-46a1-99bf-8f5f622df595-kube-api-access-lcsvs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861205 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861219 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a01e1041-e6a8-46a1-99bf-8f5f622df595-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861236 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861251 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a01e1041-e6a8-46a1-99bf-8f5f622df595-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861265 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861280 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qkxng\" (UniqueName: \"kubernetes.io/projected/b4a64170-0c8b-4f86-ba3a-e84320fa0472-kube-api-access-qkxng\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861295 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861310 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a01e1041-e6a8-46a1-99bf-8f5f622df595-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861326 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861341 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861355 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.861359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861355 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861371 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4a64170-0c8b-4f86-ba3a-e84320fa0472-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861389 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.861797 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.862035 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.862220 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.862559 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.862752 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.863052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.862888 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:05:03.863653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.863628 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.863934 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.863907 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq" pod "4117524c-c6d8-46d1-8516-9138033f2b06" (UID: "4117524c-c6d8-46d1-8516-9138033f2b06"). InnerVolumeSpecName "kube-api-access-r5csq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.864236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.864216 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2" pod "6df1d58a-ceab-439e-8fa7-7496a990fa97" (UID: "6df1d58a-ceab-439e-8fa7-7496a990fa97"). InnerVolumeSpecName "kube-api-access-prqn2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.864517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.864492 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:05:03.864865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.864846 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2" pod "f828a59b-451c-4fea-a029-c8a3af694295" (UID: "f828a59b-451c-4fea-a029-c8a3af694295"). InnerVolumeSpecName "kube-api-access-nqgw2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:05:03.867815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.867794 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 18:05:03.873012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.872988 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-752qt"] Nov 03 18:05:03.963332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963290 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prqn2\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-kube-api-access-prqn2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963324 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5csq\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-kube-api-access-r5csq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963334 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963356 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f828a59b-451c-4fea-a029-c8a3af694295-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963366 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f828a59b-451c-4fea-a029-c8a3af694295-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963375 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963383 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963395 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqgw2\" (UniqueName: \"kubernetes.io/projected/f828a59b-451c-4fea-a029-c8a3af694295-kube-api-access-nqgw2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963404 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4117524c-c6d8-46d1-8516-9138033f2b06-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963413 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4117524c-c6d8-46d1-8516-9138033f2b06-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963421 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963428 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6df1d58a-ceab-439e-8fa7-7496a990fa97-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:03.963630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:03.963436 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4117524c-c6d8-46d1-8516-9138033f2b06-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:05:04.299596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.299489 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" path="/var/lib/kubelet/pods/7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee/volumes" Nov 03 18:05:04.300138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.300118 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" path="/var/lib/kubelet/pods/e692a971-c058-4d22-8bdd-0992fef3c254/volumes" Nov 03 18:05:04.300652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.300628 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" path="/var/lib/kubelet/pods/f03b91bd-ab96-4797-ad4c-e214ba415a4a/volumes" Nov 03 18:05:04.337342 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.337314 4980 generic.go:356] "Generic (PLEG): container finished" podID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerID="ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb" exitCode=0 Nov 03 18:05:04.337468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.337377 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" Nov 03 18:05:04.337468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.337385 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" event={"ID":"b4a64170-0c8b-4f86-ba3a-e84320fa0472","Type":"ContainerDied","Data":"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb"} Nov 03 18:05:04.337468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.337433 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6" event={"ID":"b4a64170-0c8b-4f86-ba3a-e84320fa0472","Type":"ContainerDied","Data":"0f5fcd3a62b2f691b9a7fc255d1d3e1a6634357df957424edb4176b285a2758a"} Nov 03 18:05:04.337468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.337461 4980 scope.go:122] "RemoveContainer" containerID="ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb" Nov 03 18:05:04.339308 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.339283 4980 generic.go:356] "Generic (PLEG): container finished" podID="4117524c-c6d8-46d1-8516-9138033f2b06" containerID="225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76" exitCode=0 Nov 03 18:05:04.339478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.339350 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" Nov 03 18:05:04.339478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.339358 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerDied","Data":"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76"} Nov 03 18:05:04.339478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.339397 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq" event={"ID":"4117524c-c6d8-46d1-8516-9138033f2b06","Type":"ContainerDied","Data":"b1b1f46f3a79c8fa7029dbeb43f72c3f598f7344514347a1c3918fcec25ae81a"} Nov 03 18:05:04.343749 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.343730 4980 generic.go:356] "Generic (PLEG): container finished" podID="f828a59b-451c-4fea-a029-c8a3af694295" containerID="9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0" exitCode=0 Nov 03 18:05:04.343858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.343760 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" event={"ID":"f828a59b-451c-4fea-a029-c8a3af694295","Type":"ContainerDied","Data":"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0"} Nov 03 18:05:04.343858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.343813 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" event={"ID":"f828a59b-451c-4fea-a029-c8a3af694295","Type":"ContainerDied","Data":"4b47e0ede04a133c7ffd4768c381b8e8424985032278297eac0bb6b0c5688f80"} Nov 03 18:05:04.343858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.343772 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6" Nov 03 18:05:04.345225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.345203 4980 generic.go:356] "Generic (PLEG): container finished" podID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerID="ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f" exitCode=0 Nov 03 18:05:04.345337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.345244 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" Nov 03 18:05:04.345337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.345289 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" event={"ID":"a01e1041-e6a8-46a1-99bf-8f5f622df595","Type":"ContainerDied","Data":"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f"} Nov 03 18:05:04.345438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.345335 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d" event={"ID":"a01e1041-e6a8-46a1-99bf-8f5f622df595","Type":"ContainerDied","Data":"761e20a18d8e74acf3d2040c34ff70f51723d7788103f177cbafea389a8741c8"} Nov 03 18:05:04.348844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.348824 4980 generic.go:356] "Generic (PLEG): container finished" podID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerID="51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c" exitCode=0 Nov 03 18:05:04.348940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.348856 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" event={"ID":"6df1d58a-ceab-439e-8fa7-7496a990fa97","Type":"ContainerDied","Data":"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c"} Nov 03 18:05:04.348940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.348867 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" Nov 03 18:05:04.348940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.348889 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp" event={"ID":"6df1d58a-ceab-439e-8fa7-7496a990fa97","Type":"ContainerDied","Data":"9a7cda1e23d658b1485abd9897c17dd04f66d238f87781b5d13ab3f11e591663"} Nov 03 18:05:04.354469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.350933 4980 scope.go:122] "RemoveContainer" containerID="ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb" Nov 03 18:05:04.354469 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.354132 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb\": container with ID starting with ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb not found: ID does not exist" containerID="ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb" Nov 03 18:05:04.354469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.354161 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb"} err="failed to get container status \"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb\": rpc error: code = NotFound desc = could not find container \"ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb\": container with ID starting with ee99b0d929ee06a6cdfd688962c9a7adf5b6c52ef233fa0aada30dfff4ac9ebb not found: ID does not exist" Nov 03 18:05:04.354469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.354180 4980 scope.go:122] "RemoveContainer" containerID="225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76" Nov 03 18:05:04.367839 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.367818 4980 scope.go:122] "RemoveContainer" containerID="6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047" Nov 03 18:05:04.389924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.389880 4980 scope.go:122] "RemoveContainer" containerID="225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76" Nov 03 18:05:04.390641 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.390600 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76\": container with ID starting with 225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76 not found: ID does not exist" containerID="225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76" Nov 03 18:05:04.390809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.390653 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76"} err="failed to get container status \"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76\": rpc error: code = NotFound desc = could not find container \"225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76\": container with ID starting with 225c845625e3ed39b8a7d09d969456af8f449e90cef6e2dad10e2e49ed9d1d76 not found: ID does not exist" Nov 03 18:05:04.390809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.390680 4980 scope.go:122] "RemoveContainer" containerID="6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047" Nov 03 18:05:04.391160 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.391138 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047\": container with ID starting with 6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047 not found: ID does not exist" containerID="6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047" Nov 03 18:05:04.391242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.391173 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047"} err="failed to get container status \"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047\": rpc error: code = NotFound desc = could not find container \"6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047\": container with ID starting with 6aa8ba78a6339f636615cd58fcdc7de9b28632cd84e9a28df5c37d37fa70e047 not found: ID does not exist" Nov 03 18:05:04.391242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.391197 4980 scope.go:122] "RemoveContainer" containerID="9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0" Nov 03 18:05:04.403529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.403509 4980 scope.go:122] "RemoveContainer" containerID="9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0" Nov 03 18:05:04.403921 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.403895 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0\": container with ID starting with 9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0 not found: ID does not exist" containerID="9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0" Nov 03 18:05:04.404018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.403936 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0"} err="failed to get container status \"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0\": rpc error: code = NotFound desc = could not find container \"9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0\": container with ID starting with 9ab7e29f737a037ac3d792f1646dbfbb61366fcb72f3ce7f35107c5846a1ebe0 not found: ID does not exist" Nov 03 18:05:04.404018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.403953 4980 scope.go:122] "RemoveContainer" containerID="ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f" Nov 03 18:05:04.415957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.415934 4980 scope.go:122] "RemoveContainer" containerID="ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f" Nov 03 18:05:04.416313 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.416289 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f\": container with ID starting with ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f not found: ID does not exist" containerID="ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f" Nov 03 18:05:04.416412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.416324 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f"} err="failed to get container status \"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f\": rpc error: code = NotFound desc = could not find container \"ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f\": container with ID starting with ad7248b99ee2c367525d97ef99e6d818182a8aed593f9b9e06fd53863229612f not found: ID does not exist" Nov 03 18:05:04.416412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.416348 4980 scope.go:122] "RemoveContainer" containerID="51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c" Nov 03 18:05:04.429517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.429497 4980 scope.go:122] "RemoveContainer" containerID="51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c" Nov 03 18:05:04.429920 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:04.429896 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c\": container with ID starting with 51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c not found: ID does not exist" containerID="51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c" Nov 03 18:05:04.429979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.429932 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c"} err="failed to get container status \"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c\": rpc error: code = NotFound desc = could not find container \"51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c\": container with ID starting with 51523c2243ad3c003d1c7756072b6e6c57076773a9c364a7708fcad21964f02c not found: ID does not exist" Nov 03 18:05:04.639411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.639378 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 18:05:04.643160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.643128 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-c4w8d"] Nov 03 18:05:04.752170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.752139 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 18:05:04.760441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.760413 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-z6bt6"] Nov 03 18:05:04.874815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.874732 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 18:05:04.881466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:04.881435 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-kb6tp"] Nov 03 18:05:05.002719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:05.002688 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 18:05:05.008598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:05.008555 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-vrdl6"] Nov 03 18:05:05.121977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:05.121935 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 18:05:05.128503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:05.128424 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-c76280-openshift-default-6fdf6b4d79-t7snq"] Nov 03 18:05:06.296738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:06.296708 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" path="/var/lib/kubelet/pods/4117524c-c6d8-46d1-8516-9138033f2b06/volumes" Nov 03 18:05:06.297313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:06.297298 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" path="/var/lib/kubelet/pods/6df1d58a-ceab-439e-8fa7-7496a990fa97/volumes" Nov 03 18:05:06.297810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:06.297795 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" path="/var/lib/kubelet/pods/a01e1041-e6a8-46a1-99bf-8f5f622df595/volumes" Nov 03 18:05:06.298645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:06.298629 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" path="/var/lib/kubelet/pods/b4a64170-0c8b-4f86-ba3a-e84320fa0472/volumes" Nov 03 18:05:06.299112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:06.299097 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f828a59b-451c-4fea-a029-c8a3af694295" path="/var/lib/kubelet/pods/f828a59b-451c-4fea-a029-c8a3af694295/volumes" Nov 03 18:05:10.304792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.304765 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:05:10.673750 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.673671 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:05:10.674444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674387 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" Nov 03 18:05:10.674444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674411 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674446 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674456 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674471 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674477 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674489 4980 cpu_manager.go:414] "containerMap: removing container" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674495 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674510 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674516 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674529 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674535 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674551 4980 cpu_manager.go:414] "containerMap: removing container" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674557 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674612 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674619 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674652 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" Nov 03 18:05:10.674684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674659 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674837 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a01e1041-e6a8-46a1-99bf-8f5f622df595" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674854 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674864 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e692a971-c058-4d22-8bdd-0992fef3c254" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674875 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f828a59b-451c-4fea-a029-c8a3af694295" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674886 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4117524c-c6d8-46d1-8516-9138033f2b06" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674897 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ada2ca0-4a3a-4b35-b97f-a13ac669f3ee" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674912 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f03b91bd-ab96-4797-ad4c-e214ba415a4a" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674925 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4a64170-0c8b-4f86-ba3a-e84320fa0472" containerName="istio-proxy" Nov 03 18:05:10.675599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.674936 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6df1d58a-ceab-439e-8fa7-7496a990fa97" containerName="istio-proxy" Nov 03 18:05:10.679102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.679054 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.734895 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.734868 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w6-ad914d-serving-cert\"" Nov 03 18:05:10.744017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.743989 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:05:10.762346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.762312 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.762528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.762360 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5l6b\" (UniqueName: \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.762528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.762511 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.762799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.762779 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.864470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.864435 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.864683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.864533 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.864683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.864562 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5l6b\" (UniqueName: \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.864683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.864653 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.864876 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:10.864850 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-serving-cert: secret "maas-api-e2e-shared-a-w6-ad914d-serving-cert" not found Nov 03 18:05:10.864944 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:10.864931 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls podName:0a0fadc6-b470-4f2a-ae09-889668bfcdc3 nodeName:}" failed. No retries permitted until 2026-11-03 18:05:11.364915114 +0000 UTC m=+1891.193998719 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls") pod "maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" (UID: "0a0fadc6-b470-4f2a-ae09-889668bfcdc3") : secret "maas-api-e2e-shared-a-w6-ad914d-serving-cert" not found Nov 03 18:05:10.865214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.865194 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.867645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.867619 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:10.975597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:10.972709 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5l6b\" (UniqueName: \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:11.403724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:11.403632 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:11.406222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:11.406197 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:11.593016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:11.592983 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:11.979900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:11.979872 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:05:11.980746 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:05:11.980713 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a0fadc6_b470_4f2a_ae09_889668bfcdc3.slice/crio-ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69 WatchSource:0}: Error finding container ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69: Status 404 returned error can't find the container with id ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69 Nov 03 18:05:11.982307 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:11.982290 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:05:12.413035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:12.412951 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" event={"ID":"0a0fadc6-b470-4f2a-ae09-889668bfcdc3","Type":"ContainerStarted","Data":"ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69"} Nov 03 18:05:13.421202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:13.421167 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" event={"ID":"0a0fadc6-b470-4f2a-ae09-889668bfcdc3","Type":"ContainerStarted","Data":"00ea6009572e958a1885efc070f9f9688794c6d6e74ff2025c862738c836e034"} Nov 03 18:05:13.421659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:13.421238 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:13.545436 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:13.545379 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" podStartSLOduration=2.653585601 podStartE2EDuration="3.545364485s" totalImagesPullingTime="891.778884ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:13.543339795 +0000 UTC m=+1893.372423400" watchObservedRunningTime="2026-11-03 18:05:13.545364485 +0000 UTC m=+1893.374448087" Nov 03 18:05:14.521050 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:14.521012 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:05:15.353164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.353131 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:05:15.356707 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.356684 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.394106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.394065 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.394262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.394195 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.394334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.394318 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.394470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.394412 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.394910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.394888 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.395095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.395077 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmqsb\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.395241 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.395223 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.395545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.395515 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.395781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.395756 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.417303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.417269 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-b05459-openshift-default-dockercfg-p7682\"" Nov 03 18:05:15.471793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.471754 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.533651 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.533750 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.533857 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.533921 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.533990 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.534034 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmqsb\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.534061 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.534164 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.534201 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.535186 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.535482 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.535974 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.537160 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.537648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.537417 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.568526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.556143 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.607448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.606424 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.644596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.618277 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmqsb\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.650130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.645323 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\") pod \"e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.672389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.672357 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:15.974258 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.974217 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:05:15.979210 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:15.979180 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.035410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.035355 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-d9a300-serving-cert\"" Nov 03 18:05:16.040123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.040089 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:05:16.160442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.160405 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzvqt\" (UniqueName: \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.160652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.160448 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.160721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.160698 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.160818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.160801 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.198134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.198006 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:05:16.199236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.199202 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:05:16.199343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.199281 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:05:16.199343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.199326 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.264621 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.264720 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.264835 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzvqt\" (UniqueName: \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.264869 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:16.267410 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-serving-cert: secret "maas-api-e2e-shared-a-w2-d9a300-serving-cert" not found Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:16.267489 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls podName:eb441d60-e513-41d0-b889-17964b4d2ac3 nodeName:}" failed. No retries permitted until 2026-11-03 18:05:16.767467419 +0000 UTC m=+1896.596551015 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls") pod "maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" (UID: "eb441d60-e513-41d0-b889-17964b4d2ac3") : secret "maas-api-e2e-shared-a-w2-d9a300-serving-cert" not found Nov 03 18:05:16.270640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.268469 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.301420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.300283 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.362150 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.362076 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzvqt\" (UniqueName: \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.467225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.467156 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerStarted","Data":"93c4a949923fd38396a183373b9995a3ae2b041b30f6cfdb9964997f3ee186bb"} Nov 03 18:05:16.784594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.780189 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.818593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.813081 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:16.895894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:16.895623 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:17.496813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:17.496492 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:05:17.540587 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:17.540526 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerStarted","Data":"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c"} Nov 03 18:05:17.668386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:17.668319 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podStartSLOduration=2.6683046 podStartE2EDuration="2.6683046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:17.665767538 +0000 UTC m=+1897.494851140" watchObservedRunningTime="2026-11-03 18:05:17.6683046 +0000 UTC m=+1897.497388201" Nov 03 18:05:17.673090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:17.673058 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:17.679845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:17.679816 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:18.659610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:18.658917 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" event={"ID":"eb441d60-e513-41d0-b889-17964b4d2ac3","Type":"ContainerStarted","Data":"9172cf0b1dfad537e7b12e7a2f7e74ef9c2cc1c1bef45538750770efb6d4ba9b"} Nov 03 18:05:18.659610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:18.658993 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:18.695948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:18.695920 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:05:19.394653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.394280 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:05:19.426598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.395321 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:05:19.497598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.496716 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Nov 03 18:05:19.500322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.500295 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:05:19.817705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.817652 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" event={"ID":"eb441d60-e513-41d0-b889-17964b4d2ac3","Type":"ContainerStarted","Data":"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9"} Nov 03 18:05:19.818434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:19.818411 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:20.100288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:20.100167 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" podStartSLOduration=3.918464761 podStartE2EDuration="5.100144305s" totalImagesPullingTime="1.181679544s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:20.05288022 +0000 UTC m=+1899.881963823" watchObservedRunningTime="2026-11-03 18:05:20.100144305 +0000 UTC m=+1899.929227909" Nov 03 18:05:20.234601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:20.224708 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:05:20.262472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:20.232646 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": context deadline exceeded" Nov 03 18:05:25.885816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:25.885765 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:05:26.175529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.175455 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:05:26.180595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.180549 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.237439 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.237404 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-b05459\"" Nov 03 18:05:26.237673 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.237556 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-b05459-dockercfg-xr9mg\"" Nov 03 18:05:26.241134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.241107 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:05:26.439655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.400174 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnlgn\" (UniqueName: \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.440166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.440126 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.560592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.541768 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnlgn\" (UniqueName: \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.565287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.565239 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.565870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.564791 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:05:26.566406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.566363 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.571914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.571870 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:05:26.572262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.572183 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.624072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.624040 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnlgn\" (UniqueName: \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\") pod \"payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.667321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.667290 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ddtt\" (UniqueName: \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.667531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.667509 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.768793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.768699 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.768956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.768842 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ddtt\" (UniqueName: \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.769409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.769388 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.795807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.795777 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:26.835709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.835674 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ddtt\" (UniqueName: \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\") pod \"payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:26.897140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:26.897104 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:27.192272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.192241 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:05:27.194295 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:05:27.194265 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfaf9cfee_296e_4047_be51_60d31158f224.slice/crio-70a04b34beab80495d57dcaecba741c708be68a76ad8120ebe4c89edb089858d WatchSource:0}: Error finding container 70a04b34beab80495d57dcaecba741c708be68a76ad8120ebe4c89edb089858d: Status 404 returned error can't find the container with id 70a04b34beab80495d57dcaecba741c708be68a76ad8120ebe4c89edb089858d Nov 03 18:05:27.306225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.306197 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:05:27.306938 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:05:27.306914 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa811b8a_06dd_461a_b181_f67deb1780f6.slice/crio-49f3e5572269213fd971206a80da433fcc4c7ad312dd6e45f4b74f2bd377bdbe WatchSource:0}: Error finding container 49f3e5572269213fd971206a80da433fcc4c7ad312dd6e45f4b74f2bd377bdbe: Status 404 returned error can't find the container with id 49f3e5572269213fd971206a80da433fcc4c7ad312dd6e45f4b74f2bd377bdbe Nov 03 18:05:27.955878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.955841 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" event={"ID":"faf9cfee-296e-4047-be51-60d31158f224","Type":"ContainerStarted","Data":"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad"} Nov 03 18:05:27.956367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.955902 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" event={"ID":"faf9cfee-296e-4047-be51-60d31158f224","Type":"ContainerStarted","Data":"70a04b34beab80495d57dcaecba741c708be68a76ad8120ebe4c89edb089858d"} Nov 03 18:05:27.956367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.956033 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:27.957556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.957532 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" event={"ID":"fa811b8a-06dd-461a-b181-f67deb1780f6","Type":"ContainerStarted","Data":"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f"} Nov 03 18:05:27.957711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.957608 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" event={"ID":"fa811b8a-06dd-461a-b181-f67deb1780f6","Type":"ContainerStarted","Data":"49f3e5572269213fd971206a80da433fcc4c7ad312dd6e45f4b74f2bd377bdbe"} Nov 03 18:05:27.957711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:27.957641 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:28.082815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:28.082742 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" podStartSLOduration=2.082723744 podStartE2EDuration="2.082723744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.077363972 +0000 UTC m=+1907.906447573" watchObservedRunningTime="2026-11-03 18:05:28.082723744 +0000 UTC m=+1907.911807351" Nov 03 18:05:28.202599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:28.201251 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" podStartSLOduration=2.201233959 podStartE2EDuration="2.201233959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:05:28.200298494 +0000 UTC m=+1908.029382098" watchObservedRunningTime="2026-11-03 18:05:28.201233959 +0000 UTC m=+1908.030317563" Nov 03 18:05:38.984276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:38.982462 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:05:38.984276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:05:38.983622 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:05:44.522752 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:05:44.522712 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:06:03.970641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:03.969608 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:06:03.978778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:03.978745 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.056595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.055594 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:06:04.108598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.103959 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.108598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.104108 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npxhb\" (UniqueName: \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.108598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.104301 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.108598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.104352 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.108598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.104749 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-8dfc64-serving-cert\"" Nov 03 18:06:04.511658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.508856 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.511658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.508939 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.511658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.509102 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.511658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.509226 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npxhb\" (UniqueName: \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.798747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.798665 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.851180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.851130 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:04.980598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.955754 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-npxhb\" (UniqueName: \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:05.024161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:04.984060 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:05.221887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:05.221848 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:06.190009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:06.072329 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:06:06.346645 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:06.225947 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod633a5340_a940_4657_b45f_16a2dddf9233.slice/crio-ccb15e07134b8c9ebda64eaff02beb1305b544fb39bc5110936c76b80c13838f WatchSource:0}: Error finding container ccb15e07134b8c9ebda64eaff02beb1305b544fb39bc5110936c76b80c13838f: Status 404 returned error can't find the container with id ccb15e07134b8c9ebda64eaff02beb1305b544fb39bc5110936c76b80c13838f Nov 03 18:06:07.093598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.092670 4980 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.093598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.092921 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:06:07.205598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.183612 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.207817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.207779 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": context deadline exceeded" Nov 03 18:06:07.349647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.349485 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" probeResult="failure" output="Get \"https://10.132.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.367134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.363663 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.398808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.398752 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.411611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.402630 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:07.476943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.431084 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" event={"ID":"633a5340-a940-4657-b45f-16a2dddf9233","Type":"ContainerStarted","Data":"ccb15e07134b8c9ebda64eaff02beb1305b544fb39bc5110936c76b80c13838f"} Nov 03 18:06:07.476943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:07.457219 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded" Nov 03 18:06:08.153698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.116281 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": context deadline exceeded" Nov 03 18:06:08.153698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.116925 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:06:08.177636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.177266 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:06:08.177636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.177318 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.190061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.190013 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.190740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.190714 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:06:08.241899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.241862 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.266975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.242242 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" probeResult="failure" output="Get \"https://10.132.0.67:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.267522 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.267496 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:06:08.386632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.386530 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": context deadline exceeded" Nov 03 18:06:08.408262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.408160 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" podUID="bf5ad4c1-ae4c-487b-baa4-bf5385b1420e" containerName="main" probeResult="failure" output="Get \"https://10.132.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.456613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.426280 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:08.488866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:08.460688 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:09.016218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.016180 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:06:09.016628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.016552 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:06:09.022112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.022091 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:06:09.023286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.023267 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:06:09.024595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.024549 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:06:09.024802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.024786 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:06:09.170596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.168709 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" probeResult="failure" output="Get \"https://10.132.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Nov 03 18:06:09.231704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.231396 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:06:09.231704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.231455 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:06:09.263619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.254720 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:06:09.264126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.263661 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:06:09.313343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:09.313211 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj" Nov 03 18:06:10.026618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:10.026543 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" event={"ID":"633a5340-a940-4657-b45f-16a2dddf9233","Type":"ContainerStarted","Data":"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908"} Nov 03 18:06:10.026896 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:10.026657 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:10.403396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:10.403280 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" podStartSLOduration=6.298029873 podStartE2EDuration="7.403264254s" totalImagesPullingTime="1.105234381s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:10.402621981 +0000 UTC m=+1950.231705585" watchObservedRunningTime="2026-11-03 18:06:10.403264254 +0000 UTC m=+1950.232347860" Nov 03 18:06:11.547233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.547199 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:06:11.552604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.552565 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.599123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.599086 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:06:11.604962 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.604925 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.612191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.612168 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-ad914d-openshift-default-dockercfg-9xgcc\"" Nov 03 18:06:11.637807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.637768 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.637837 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.637961 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638010 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638223 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638201 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638254 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638300 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638399 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638440 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638622 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.638701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638665 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638706 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zwgm\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638810 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638881 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638857 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638891 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.638987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.638971 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.639091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.639074 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.639134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.639122 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgvn5\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.657680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.657645 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:06:11.709082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.709043 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:06:11.740613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740555 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.740818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740634 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.740818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740663 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.740818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740696 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.740818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740781 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740819 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgvn5\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740852 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740898 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740942 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.740983 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741066 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741089 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741137 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741184 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741219 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741255 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741419 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.741638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741482 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.741638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.741544 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zwgm\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.742255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.742226 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.742796 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.742727 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.742910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.742885 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.743550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.743249 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.743550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.743295 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.743550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.743502 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.743786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.743737 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.744035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.743993 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.744599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.744261 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.744599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.744330 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.746487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.746451 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.746855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.746834 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.746960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.746937 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.806319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.806138 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgvn5\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.806489 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.806360 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.806619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.806596 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zwgm\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.806691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.806616 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\") pod \"e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:11.870084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.870006 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:11.923332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:11.923284 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:12.316886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.316852 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:06:12.332319 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:12.332291 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7909f6_bccc_462b_9c42_fc120e4dc653.slice/crio-e3503dd7c3260dc5715a82a8c77d67444eb0232b8683848325d2d4d13730dcbe WatchSource:0}: Error finding container e3503dd7c3260dc5715a82a8c77d67444eb0232b8683848325d2d4d13730dcbe: Status 404 returned error can't find the container with id e3503dd7c3260dc5715a82a8c77d67444eb0232b8683848325d2d4d13730dcbe Nov 03 18:06:12.335514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.335477 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:12.335646 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.335560 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:12.335717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.335655 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:12.370816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.370792 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:06:12.373982 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:12.373956 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd2f3bdf_65d7_4bf2_9d75_d1e6dd066ec9.slice/crio-0f819abe0a2a249952db94c9c97c0b2d54b4deb9e255470f9e7a35d2aec15549 WatchSource:0}: Error finding container 0f819abe0a2a249952db94c9c97c0b2d54b4deb9e255470f9e7a35d2aec15549: Status 404 returned error can't find the container with id 0f819abe0a2a249952db94c9c97c0b2d54b4deb9e255470f9e7a35d2aec15549 Nov 03 18:06:12.376731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.376702 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:12.376828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.376764 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:12.376828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:12.376796 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:13.058113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.058059 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" event={"ID":"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9","Type":"ContainerStarted","Data":"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3"} Nov 03 18:06:13.058642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.058135 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" event={"ID":"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9","Type":"ContainerStarted","Data":"0f819abe0a2a249952db94c9c97c0b2d54b4deb9e255470f9e7a35d2aec15549"} Nov 03 18:06:13.059793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.059764 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" event={"ID":"7e7909f6-bccc-462b-9c42-fc120e4dc653","Type":"ContainerStarted","Data":"a1210767430e43378e0878c687aa1b773cd459dedefc8e0bbb1427e1a4593128"} Nov 03 18:06:13.059922 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.059816 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" event={"ID":"7e7909f6-bccc-462b-9c42-fc120e4dc653","Type":"ContainerStarted","Data":"e3503dd7c3260dc5715a82a8c77d67444eb0232b8683848325d2d4d13730dcbe"} Nov 03 18:06:13.184339 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.184281 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podStartSLOduration=2.184267142 podStartE2EDuration="2.184267142s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.180648939 +0000 UTC m=+1953.009732540" watchObservedRunningTime="2026-11-03 18:06:13.184267142 +0000 UTC m=+1953.013350738" Nov 03 18:06:13.304253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.304191 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podStartSLOduration=2.304173783 podStartE2EDuration="2.304173783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:13.303460692 +0000 UTC m=+1953.132544295" watchObservedRunningTime="2026-11-03 18:06:13.304173783 +0000 UTC m=+1953.133257385" Nov 03 18:06:13.878594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.873430 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:13.878594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.873628 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:13.878594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.873693 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:13.924236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.924156 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:13.925952 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.925923 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:13.926102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:13.926000 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:14.413655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.412416 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": read tcp 10.132.0.2:40736->10.132.0.118:15021: read: connection reset by peer" start-of-body= Nov 03 18:06:14.413655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.412481 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": read tcp 10.132.0.2:40736->10.132.0.118:15021: read: connection reset by peer" Nov 03 18:06:14.551381 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:06:14.551300 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:06:14.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.879109 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:14.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.879181 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:14.924129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.924087 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:14.924297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:14.924154 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:15.123045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.123008 4980 generic.go:356] "Generic (PLEG): container finished" podID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerID="e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e" exitCode=0 Nov 03 18:06:15.123235 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.123085 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerDied","Data":"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e"} Nov 03 18:06:15.123913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.123891 4980 scope.go:122] "RemoveContainer" containerID="e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e" Nov 03 18:06:15.125422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.125390 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:15.125536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.125479 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:15.125536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.125530 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:15.873090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.873049 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.873748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.873705 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:15.933063 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.933029 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:15.933342 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:15.933313 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:16.042434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.042403 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:06:16.148495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.148414 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerStarted","Data":"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67"} Nov 03 18:06:16.169798 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.169768 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:06:16.212041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.211983 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.212221 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.212049 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:16.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.879029 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.879598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.879094 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:16.931405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.930777 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:16.931405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:16.930844 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:17.173655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.168824 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.173655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.168899 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:17.872053 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.872012 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.872420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.872380 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:17.932721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.932673 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:17.933348 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:17.933315 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:18.165035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.164937 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.165295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.165267 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:18.878596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.871984 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.878596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.872053 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" Nov 03 18:06:18.932595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.927012 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body= Nov 03 18:06:18.932595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:18.927075 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" Nov 03 18:06:19.171656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.171064 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:19.171656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.171131 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:19.883557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.883526 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:19.883931 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.883911 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:19.884733 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.884713 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:06:19.929547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.929517 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:19.929843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.929823 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:19.930693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:19.930673 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:06:20.174775 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:20.174679 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:20.175372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:20.175342 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:21.179771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:21.179727 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:21.180195 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:21.179794 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:22.167052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:22.167002 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" start-of-body= Nov 03 18:06:22.167267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:22.167074 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.118:15021/healthz/ready\": dial tcp 10.132.0.118:15021: connect: connection refused" Nov 03 18:06:23.168906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:23.168870 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:06:23.169382 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:23.169321 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:06:23.169986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:23.169964 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:06:28.450473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:28.450435 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 18:06:28.451033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:28.450742 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" containerID="cri-o://76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945" gracePeriod=30 Nov 03 18:06:28.963099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:28.963061 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 18:06:28.963351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:28.963325 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" containerID="cri-o://edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba" gracePeriod=30 Nov 03 18:06:33.755303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.755270 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 18:06:33.825332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825295 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825342 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825416 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825453 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825506 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825545 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825615 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825762 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825776 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825825 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:33.825878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825863 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\") pod \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\" (UID: \"dcbc7243-1aa6-4c57-be08-00a2988e99c1\") " Nov 03 18:06:33.826183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.825986 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:33.826183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826016 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:06:33.826183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826100 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:33.826835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826800 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.826835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826832 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.827009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826849 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.827009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826867 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.827009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.826883 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.827937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.827913 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:06:33.828074 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.828054 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:33.828184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.828160 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "kube-api-access-5nfhx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:33.828328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.828313 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token" pod "dcbc7243-1aa6-4c57-be08-00a2988e99c1" (UID: "dcbc7243-1aa6-4c57-be08-00a2988e99c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:33.928154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.928116 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.928154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.928158 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.928384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.928175 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nfhx\" (UniqueName: \"kubernetes.io/projected/dcbc7243-1aa6-4c57-be08-00a2988e99c1-kube-api-access-5nfhx\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:33.928384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:33.928187 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dcbc7243-1aa6-4c57-be08-00a2988e99c1-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.219319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.219293 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 18:06:34.232430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232353 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232454 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232486 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232542 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232628 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232639 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:34.232829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232686 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232773 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.232829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232802 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.233110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232842 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:34.233110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.232903 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\") pod \"d56a4cbc-cebf-435c-96ed-7950973ea815\" (UID: \"d56a4cbc-cebf-435c-96ed-7950973ea815\") " Nov 03 18:06:34.233110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.233027 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:34.233739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.233563 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.233739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.233622 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.233739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.233637 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.233739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.233705 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:34.234065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.234024 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:06:34.235705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.235658 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:34.236134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.236083 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "kube-api-access-dx5bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:34.236238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.236217 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:34.237903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.237879 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo" pod "d56a4cbc-cebf-435c-96ed-7950973ea815" (UID: "d56a4cbc-cebf-435c-96ed-7950973ea815"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:06:34.335089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335055 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.335089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335084 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d56a4cbc-cebf-435c-96ed-7950973ea815-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.335089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335093 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.335310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335101 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d56a4cbc-cebf-435c-96ed-7950973ea815-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.335310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335110 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d56a4cbc-cebf-435c-96ed-7950973ea815-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.335310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.335118 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dx5bf\" (UniqueName: \"kubernetes.io/projected/d56a4cbc-cebf-435c-96ed-7950973ea815-kube-api-access-dx5bf\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:34.470922 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.470890 4980 generic.go:356] "Generic (PLEG): container finished" podID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerID="76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945" exitCode=0 Nov 03 18:06:34.471105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.470957 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" Nov 03 18:06:34.471105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.470974 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" event={"ID":"dcbc7243-1aa6-4c57-be08-00a2988e99c1","Type":"ContainerDied","Data":"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945"} Nov 03 18:06:34.471105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.471027 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t" event={"ID":"dcbc7243-1aa6-4c57-be08-00a2988e99c1","Type":"ContainerDied","Data":"9121c590828df6c750618c655ed39716391508dd6ec02ecf6e76e7c4cb90ccf5"} Nov 03 18:06:34.471105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.471052 4980 scope.go:122] "RemoveContainer" containerID="76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945" Nov 03 18:06:34.472494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.472471 4980 generic.go:356] "Generic (PLEG): container finished" podID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerID="edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba" exitCode=0 Nov 03 18:06:34.472667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.472515 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" Nov 03 18:06:34.472667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.472558 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" event={"ID":"d56a4cbc-cebf-435c-96ed-7950973ea815","Type":"ContainerDied","Data":"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba"} Nov 03 18:06:34.472667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.472643 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn" event={"ID":"d56a4cbc-cebf-435c-96ed-7950973ea815","Type":"ContainerDied","Data":"ae4d1ca7f301549d8ac2238074e764a872903ac8938d51f353a2047548e02d1c"} Nov 03 18:06:34.496322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.496293 4980 scope.go:122] "RemoveContainer" containerID="76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945" Nov 03 18:06:34.501069 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:06:34.496757 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945\": container with ID starting with 76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945 not found: ID does not exist" containerID="76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945" Nov 03 18:06:34.501069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.496797 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945"} err="failed to get container status \"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945\": rpc error: code = NotFound desc = could not find container \"76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945\": container with ID starting with 76d6d26c3f2156affea21900e24f76aaead97f4abd65f0146bfd23f76de21945 not found: ID does not exist" Nov 03 18:06:34.501069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.496827 4980 scope.go:122] "RemoveContainer" containerID="edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba" Nov 03 18:06:34.516692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.516670 4980 scope.go:122] "RemoveContainer" containerID="edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba" Nov 03 18:06:34.517051 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:06:34.517026 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba\": container with ID starting with edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba not found: ID does not exist" containerID="edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba" Nov 03 18:06:34.517127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.517069 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba"} err="failed to get container status \"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba\": rpc error: code = NotFound desc = could not find container \"edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba\": container with ID starting with edcb431de3c79a7aca8640183376705478ae171bcc0d2ee8c611f8f756150fba not found: ID does not exist" Nov 03 18:06:34.642315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.642282 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 18:06:34.650204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.650162 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-v624t"] Nov 03 18:06:34.777660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.774867 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 18:06:34.783621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:34.781610 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-pcdhn"] Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.068443 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.069583 4980 cpu_manager.go:414] "containerMap: removing container" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.069608 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.069696 4980 cpu_manager.go:414] "containerMap: removing container" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.069703 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.071303 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.071329 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" containerName="istio-proxy" Nov 03 18:06:35.080720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.079964 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.112403 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.111639 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:06:35.142298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.142263 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-2569c8-openshift-default-dockercfg-zlx9p\"" Nov 03 18:06:35.160734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.160700 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.160886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.160744 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.160982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.160961 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161002 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hkq\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161078 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161153 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161215 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161268 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161252 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.161306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.161278 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.262648 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.262993 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hkq\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263054 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263149 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263158 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263253 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263296 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263324 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263416 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263441 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.263944 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.264351 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.264484 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.274085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.264798 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.298681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.294194 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.298681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.297284 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.327019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.326985 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.327500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.327473 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hkq\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\") pod \"e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.430292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.429981 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:35.892906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.892875 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:06:35.894318 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:35.894286 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f9cc605_041d_43bc_b381_ae12b74e09a3.slice/crio-f5d715e886a11e6ee50dc87504059fe068b3a6e5a2cf834f99952c260330210f WatchSource:0}: Error finding container f5d715e886a11e6ee50dc87504059fe068b3a6e5a2cf834f99952c260330210f: Status 404 returned error can't find the container with id f5d715e886a11e6ee50dc87504059fe068b3a6e5a2cf834f99952c260330210f Nov 03 18:06:35.897143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.897109 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:35.897250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.897177 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:35.897250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:35.897207 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:36.366596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:36.348103 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d56a4cbc-cebf-435c-96ed-7950973ea815" path="/var/lib/kubelet/pods/d56a4cbc-cebf-435c-96ed-7950973ea815/volumes" Nov 03 18:06:36.381652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:36.373376 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcbc7243-1aa6-4c57-be08-00a2988e99c1" path="/var/lib/kubelet/pods/dcbc7243-1aa6-4c57-be08-00a2988e99c1/volumes" Nov 03 18:06:36.502594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:36.502470 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerStarted","Data":"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194"} Nov 03 18:06:36.502594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:36.502541 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerStarted","Data":"f5d715e886a11e6ee50dc87504059fe068b3a6e5a2cf834f99952c260330210f"} Nov 03 18:06:36.634640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:36.634511 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podStartSLOduration=2.634491676 podStartE2EDuration="2.634491676s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:36.631002826 +0000 UTC m=+1976.460086437" watchObservedRunningTime="2026-11-03 18:06:36.634491676 +0000 UTC m=+1976.463575279" Nov 03 18:06:37.430719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:37.430689 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:37.436995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:37.436966 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:37.510382 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:37.510337 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:37.511806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:37.511779 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:06:40.141896 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.141856 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:06:40.164874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.164844 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.272438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.272403 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-19d275-openshift-default-dockercfg-h6mwx\"" Nov 03 18:06:40.273100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273065 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273115 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273177 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273227 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273252 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273335 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273376 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzs4c\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273410 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.273601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.273563 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.282769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.282743 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:06:40.374862 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.374827 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.374879 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.374960 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375011 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375035 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375147 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375191 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzs4c\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375245 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375455 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.375600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375468 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.376013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.375991 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.376101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.376010 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.376394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.376372 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.376488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.376395 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.377662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.377633 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.378911 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.378887 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.457669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.457641 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzs4c\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.457669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.457659 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\") pod \"e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:40.598639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.597916 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-19d275-openshift-default-dockercfg-h6mwx\"" Nov 03 18:06:40.603597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:40.601743 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:41.147484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.147457 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:06:41.148113 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:41.148094 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36b392c_adef_417f_883f_ee3d30774714.slice/crio-76bbb87dd3a12f5c929dcddcd90a42728446445b0c57aac7121abace91b33fd9 WatchSource:0}: Error finding container 76bbb87dd3a12f5c929dcddcd90a42728446445b0c57aac7121abace91b33fd9: Status 404 returned error can't find the container with id 76bbb87dd3a12f5c929dcddcd90a42728446445b0c57aac7121abace91b33fd9 Nov 03 18:06:41.151714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.151677 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:41.151823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.151766 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:41.151823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.151801 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:41.565194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.565155 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerStarted","Data":"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111"} Nov 03 18:06:41.565434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.565212 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerStarted","Data":"76bbb87dd3a12f5c929dcddcd90a42728446445b0c57aac7121abace91b33fd9"} Nov 03 18:06:41.602800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.602768 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:41.743145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:41.743084 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podStartSLOduration=1.743069312 podStartE2EDuration="1.743069312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:41.738456593 +0000 UTC m=+1981.567540195" watchObservedRunningTime="2026-11-03 18:06:41.743069312 +0000 UTC m=+1981.572152914" Nov 03 18:06:42.608157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:42.608126 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:43.579511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:43.579479 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:43.580361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:43.580340 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:06:44.521057 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:06:44.521027 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:06:44.528458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:44.528427 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 18:06:44.528724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:44.528692 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" containerID="cri-o://438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18" gracePeriod=30 Nov 03 18:06:45.508429 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.508379 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:06:45.541559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.541524 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.606946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.606914 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w6-2569c8-dockercfg-w82xf\"" Nov 03 18:06:45.607930 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.607902 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w6-2569c8\"" Nov 03 18:06:45.641649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.624944 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.642390 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.642171 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxdpd\" (UniqueName: \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.733558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.733493 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:06:45.903898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.903776 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxdpd\" (UniqueName: \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.904650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.904557 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.928763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.919152 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.938946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.938611 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:06:45.951410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.951376 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:45.955613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.955583 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:06:45.984204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.984171 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxdpd\" (UniqueName: \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\") pod \"payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:45.996148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:45.996119 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:46.123361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.123323 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.123547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.123429 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s87l\" (UniqueName: \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.254683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.227630 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s87l\" (UniqueName: \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.254683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.227926 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.254683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.228894 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.443524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.443492 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s87l\" (UniqueName: \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\") pod \"payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.469898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.458099 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:06:46.572843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.572642 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:46.989595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.983373 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" event={"ID":"571911b9-e1e4-40e8-8cea-cdfe688d7a27","Type":"ContainerStarted","Data":"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67"} Nov 03 18:06:46.989595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.983453 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" event={"ID":"571911b9-e1e4-40e8-8cea-cdfe688d7a27","Type":"ContainerStarted","Data":"4f4baf3b1105fb3d519699764fb7586d2213eaf2f8502808fd465c8ba79382dc"} Nov 03 18:06:46.989595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:46.984640 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:47.045283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:47.041753 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:06:47.128795 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:47.128739 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" podStartSLOduration=2.128719976 podStartE2EDuration="2.128719976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:47.120956644 +0000 UTC m=+1986.950040287" watchObservedRunningTime="2026-11-03 18:06:47.128719976 +0000 UTC m=+1986.957803579" Nov 03 18:06:47.994005 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:47.993942 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" event={"ID":"f3a90b95-733c-4c85-937e-afa464c60bc5","Type":"ContainerStarted","Data":"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da"} Nov 03 18:06:47.994437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:47.994018 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" event={"ID":"f3a90b95-733c-4c85-937e-afa464c60bc5","Type":"ContainerStarted","Data":"7f5f1a6e91057dff22ae78b6108d34d3841078efbab9e538d3cbc84da4894bcf"} Nov 03 18:06:48.127819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:48.127747 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" podStartSLOduration=3.12772788 podStartE2EDuration="3.12772788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:48.120955302 +0000 UTC m=+1987.950038915" watchObservedRunningTime="2026-11-03 18:06:48.12772788 +0000 UTC m=+1987.956811483" Nov 03 18:06:49.009288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:49.009236 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:06:49.902536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:49.902501 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 18:06:49.999961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:49.999924 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:49.999975 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000072 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000230 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000285 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000316 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000360 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:50.000453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000373 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:06:50.000453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000371 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000469 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.000652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000506 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:50.000652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000548 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:50.000652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000599 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\") pod \"776e5da1-9243-4086-8032-1951428d3b67\" (UID: \"776e5da1-9243-4086-8032-1951428d3b67\") " Nov 03 18:06:50.001108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.000918 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:50.001624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.001555 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.001744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.001654 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.001744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.001671 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/776e5da1-9243-4086-8032-1951428d3b67-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.001744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.001685 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.001744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.001699 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.002670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.002645 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:06:50.002919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.002888 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:50.003022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.002957 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:06:50.003022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.002960 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7" pod "776e5da1-9243-4086-8032-1951428d3b67" (UID: "776e5da1-9243-4086-8032-1951428d3b67"). InnerVolumeSpecName "kube-api-access-mbwh7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:06:50.018277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.018251 4980 generic.go:356] "Generic (PLEG): container finished" podID="776e5da1-9243-4086-8032-1951428d3b67" containerID="438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18" exitCode=0 Nov 03 18:06:50.018722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.018300 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" Nov 03 18:06:50.018722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.018330 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" event={"ID":"776e5da1-9243-4086-8032-1951428d3b67","Type":"ContainerDied","Data":"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18"} Nov 03 18:06:50.018722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.018436 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx" event={"ID":"776e5da1-9243-4086-8032-1951428d3b67","Type":"ContainerDied","Data":"c28cd0b65bd9b0b2e06d8befcf3a06f711f623ccd46b06670628559341c99397"} Nov 03 18:06:50.018722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.018470 4980 scope.go:122] "RemoveContainer" containerID="438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18" Nov 03 18:06:50.037014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.036988 4980 scope.go:122] "RemoveContainer" containerID="438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18" Nov 03 18:06:50.037352 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:06:50.037331 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18\": container with ID starting with 438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18 not found: ID does not exist" containerID="438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18" Nov 03 18:06:50.037404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.037362 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18"} err="failed to get container status \"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18\": rpc error: code = NotFound desc = could not find container \"438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18\": container with ID starting with 438d952c9c8e4db519057ce905c2c79bd6d1f369f7741ea1c60b3b26447ced18 not found: ID does not exist" Nov 03 18:06:50.110209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.110115 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.110209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.110155 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mbwh7\" (UniqueName: \"kubernetes.io/projected/776e5da1-9243-4086-8032-1951428d3b67-kube-api-access-mbwh7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.110209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.110174 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/776e5da1-9243-4086-8032-1951428d3b67-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.110209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.110189 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/776e5da1-9243-4086-8032-1951428d3b67-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:06:50.207226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.207194 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 18:06:50.213269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.213242 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-2bhfx"] Nov 03 18:06:50.297300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:50.297265 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="776e5da1-9243-4086-8032-1951428d3b67" path="/var/lib/kubelet/pods/776e5da1-9243-4086-8032-1951428d3b67/volumes" Nov 03 18:06:55.339273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.315987 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:06:55.339273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.317325 4980 cpu_manager.go:414] "containerMap: removing container" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" Nov 03 18:06:55.339273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.317349 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" Nov 03 18:06:55.339273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.317524 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="776e5da1-9243-4086-8032-1951428d3b67" containerName="istio-proxy" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.376384 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.376544 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.432001 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.445558 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskcx\" (UniqueName: \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.445810 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.461888 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.489831 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-19d275-dockercfg-9b6dh\"" Nov 03 18:06:55.545276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.489917 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-19d275\"" Nov 03 18:06:55.569868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.569807 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:06:55.598380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.598281 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskcx\" (UniqueName: \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.598586 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.598545 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vh2g7\" (UniqueName: \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.598692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.598640 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.598692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.598682 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.600625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.600593 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.663767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.663726 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskcx\" (UniqueName: \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\") pod \"payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.700186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.700151 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vh2g7\" (UniqueName: \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.700367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.700202 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.700978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.700959 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.764095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.764058 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vh2g7\" (UniqueName: \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\") pod \"payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:55.949285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.949253 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:55.981992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:55.981934 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:56.363128 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:56.363086 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:06:56.364947 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:06:56.364921 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea890b4_d192_4ac3_844c_f9b8f9cdacca.slice/crio-69ca5434c35076c0223bd46f0a6329e43a78a5af5609ed482f3c46426a75b1a2 WatchSource:0}: Error finding container 69ca5434c35076c0223bd46f0a6329e43a78a5af5609ed482f3c46426a75b1a2: Status 404 returned error can't find the container with id 69ca5434c35076c0223bd46f0a6329e43a78a5af5609ed482f3c46426a75b1a2 Nov 03 18:06:56.428085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:56.428026 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:06:57.120384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.120345 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" event={"ID":"fea890b4-d192-4ac3-844c-f9b8f9cdacca","Type":"ContainerStarted","Data":"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d"} Nov 03 18:06:57.120659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.120407 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" event={"ID":"fea890b4-d192-4ac3-844c-f9b8f9cdacca","Type":"ContainerStarted","Data":"69ca5434c35076c0223bd46f0a6329e43a78a5af5609ed482f3c46426a75b1a2"} Nov 03 18:06:57.120659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.120435 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:06:57.122365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.122335 4980 generic.go:356] "Generic (PLEG): container finished" podID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerID="7da59dedf7ce897534bc5d6fce1d53f480b52cf0c4299ad6b848d5a6282af055" exitCode=0 Nov 03 18:06:57.122516 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.122367 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerDied","Data":"7da59dedf7ce897534bc5d6fce1d53f480b52cf0c4299ad6b848d5a6282af055"} Nov 03 18:06:57.122831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.122811 4980 scope.go:122] "RemoveContainer" containerID="7da59dedf7ce897534bc5d6fce1d53f480b52cf0c4299ad6b848d5a6282af055" Nov 03 18:06:57.124407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124378 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" event={"ID":"bb0d75fc-5701-4616-b748-bbaba2b388ee","Type":"ContainerStarted","Data":"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134"} Nov 03 18:06:57.124924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124423 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" event={"ID":"bb0d75fc-5701-4616-b748-bbaba2b388ee","Type":"ContainerStarted","Data":"f7b13b0cb4ea84dc138220efb39e1bdc22d71ec398f64e197440a3428b54a0aa"} Nov 03 18:06:57.124924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124522 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:06:57.124924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124840 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:57.125067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124921 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:57.125067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.124964 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:06:57.375651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.373883 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" podStartSLOduration=2.373858875 podStartE2EDuration="2.373858875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:57.241651101 +0000 UTC m=+1997.070734705" watchObservedRunningTime="2026-11-03 18:06:57.373858875 +0000 UTC m=+1997.202942478" Nov 03 18:06:57.546605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:57.543201 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" podStartSLOduration=2.543178002 podStartE2EDuration="2.543178002s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:06:57.388869207 +0000 UTC m=+1997.217952810" watchObservedRunningTime="2026-11-03 18:06:57.543178002 +0000 UTC m=+1997.372261607" Nov 03 18:06:58.134478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:58.134445 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerStarted","Data":"31b9ae168fcd3848b7bb63a3e3a27aaf5afc3efbeb2435e0e63bd8820b43fe11"} Nov 03 18:06:58.150428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:58.150405 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:06:58.150605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:58.150544 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:06:58.150669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:58.150601 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:06:59.013401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:59.013367 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:06:59.150608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:59.150556 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:06:59.150821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:06:59.150626 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:07:00.022448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:00.022416 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:07:00.150365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:00.150333 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:07:00.150539 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:00.150396 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:07:01.150550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:01.150515 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:07:01.150993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:01.150585 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:07:02.151181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:02.151145 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:07:02.151668 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:02.151198 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:07:03.150971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:03.150938 4980 patch_prober.go:29] interesting pod/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" start-of-body= Nov 03 18:07:03.151169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:03.151006 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.121:15021/healthz/ready\": dial tcp 10.132.0.121:15021: connect: connection refused" Nov 03 18:07:04.270604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:04.269797 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:07:04.280608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:04.272009 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:07:04.280852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:04.280791 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:07:08.137699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:08.137669 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:07:08.138108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:08.137752 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:07:14.521508 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:07:14.521470 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:07:16.274836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.274791 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:07:16.282346 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.281384 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.376637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.376599 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7vmc\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.377045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.377023 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.377660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.377537 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.377987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.377970 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.378295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.378278 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.378731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.378713 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.379010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.378993 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.379349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.379333 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.379701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.379545 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.413633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.392535 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:07:16.439600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.437788 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c17c6a-openshift-default-dockercfg-76mvk\"" Nov 03 18:07:16.504614 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.504549 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.504917 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.504893 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.505121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.505106 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.505352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.505331 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.505777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.505754 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.506029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.506009 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.506146 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.506132 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.506408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.506392 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7vmc\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.506597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.506563 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.507294 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.507273 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.521607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.510687 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.521607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.511751 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.521607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.519962 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.521607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.520203 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.588287 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.584189 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.593256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.593148 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7vmc\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.648443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.648370 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.650137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.650103 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\") pod \"e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:16.901799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:16.901724 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:17.349920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.349890 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:07:17.351622 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:07:17.351551 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07531d91_a805_4751_aa5e_5aca33cbccdc.slice/crio-812ebb4c10d37c9ad75f5cfbdf31821b13e79398922d345bd8671f914f54855d WatchSource:0}: Error finding container 812ebb4c10d37c9ad75f5cfbdf31821b13e79398922d345bd8671f914f54855d: Status 404 returned error can't find the container with id 812ebb4c10d37c9ad75f5cfbdf31821b13e79398922d345bd8671f914f54855d Nov 03 18:07:17.354889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.354856 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:17.355002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.354934 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:17.355002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.354981 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:17.468276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.468231 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerStarted","Data":"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408"} Nov 03 18:07:17.468457 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.468297 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerStarted","Data":"812ebb4c10d37c9ad75f5cfbdf31821b13e79398922d345bd8671f914f54855d"} Nov 03 18:07:17.902198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:17.902167 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:18.909282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:18.909250 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:19.087758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:19.087698 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podStartSLOduration=3.087681582 podStartE2EDuration="3.087681582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:17.593215582 +0000 UTC m=+2017.422299190" watchObservedRunningTime="2026-11-03 18:07:19.087681582 +0000 UTC m=+2018.916765184" Nov 03 18:07:19.485365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:19.485325 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:19.486276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:19.486256 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:07:19.701751 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:19.701711 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:07:19.701957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:19.701774 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:21.501332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.501302 4980 generic.go:356] "Generic (PLEG): container finished" podID="15237981-a676-4297-a2a2-4964991a41cf" containerID="b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c" exitCode=0 Nov 03 18:07:21.501833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.501375 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerDied","Data":"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c"} Nov 03 18:07:21.501906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.501890 4980 scope.go:122] "RemoveContainer" containerID="b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c" Nov 03 18:07:21.503523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.503497 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:21.503608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.503564 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:21.503650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:21.503621 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:22.511970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:22.511924 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerStarted","Data":"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7"} Nov 03 18:07:22.675598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:22.674098 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:07:22.675598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:22.674304 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:22.675598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:22.674348 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:23.674595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:23.673934 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:23.674595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:23.673995 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:24.497597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.492125 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:07:24.670609 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.669805 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:24.676665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.676631 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:24.677206 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.677172 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:24.730542 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.730507 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-c17c6a\"" Nov 03 18:07:24.775651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.764228 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:07:24.806632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.781061 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-c17c6a-dockercfg-fv4vz\"" Nov 03 18:07:24.843763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.840450 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvl4t\" (UniqueName: \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:24.843763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.840646 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:24.942077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.942041 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:24.942267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.942255 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvl4t\" (UniqueName: \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:24.942795 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:24.942776 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:25.005395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.005362 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvl4t\" (UniqueName: \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\") pod \"payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:25.009140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.009118 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:25.422078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.422049 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:07:25.422640 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:07:25.422610 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07107523_9d65_4545_a3f6_26474bf83639.slice/crio-68b2a72e70c68760e0e4513bdd92fd7f6d7f7bef1f9419df313f231e39445bf3 WatchSource:0}: Error finding container 68b2a72e70c68760e0e4513bdd92fd7f6d7f7bef1f9419df313f231e39445bf3: Status 404 returned error can't find the container with id 68b2a72e70c68760e0e4513bdd92fd7f6d7f7bef1f9419df313f231e39445bf3 Nov 03 18:07:25.675928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.675835 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:25.676184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.676152 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:25.694979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.694934 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" event={"ID":"07107523-9d65-4545-a3f6-26474bf83639","Type":"ContainerStarted","Data":"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61"} Nov 03 18:07:25.695442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.694999 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" event={"ID":"07107523-9d65-4545-a3f6-26474bf83639","Type":"ContainerStarted","Data":"68b2a72e70c68760e0e4513bdd92fd7f6d7f7bef1f9419df313f231e39445bf3"} Nov 03 18:07:25.695442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.695021 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:25.813497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:25.813434 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" podStartSLOduration=1.813419236 podStartE2EDuration="1.813419236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:25.813331196 +0000 UTC m=+2025.642414797" watchObservedRunningTime="2026-11-03 18:07:25.813419236 +0000 UTC m=+2025.642502839" Nov 03 18:07:26.674596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:26.673949 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:26.674596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:26.674019 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:27.673983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:27.673943 4980 patch_prober.go:29] interesting pod/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" start-of-body= Nov 03 18:07:27.674620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:27.674558 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.124:15021/healthz/ready\": dial tcp 10.132.0.124:15021: connect: connection refused" Nov 03 18:07:28.677992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:28.677959 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:07:28.678554 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:28.678449 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:07:28.679154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:28.679135 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:07:36.247061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:36.241691 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:07:36.247061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:36.241756 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:07:36.722598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:36.721780 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:07:37.621882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.621848 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:07:37.644000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.643962 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.755815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.755780 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-528e53-openshift-default-dockercfg-wvqk7\"" Nov 03 18:07:37.774598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.757080 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.810840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.810792 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qkm7\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.811254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.811222 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.811428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.811407 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.811721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.811702 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.811885 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.811865 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.851593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.845737 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.851593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.845863 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.851593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.845894 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.933095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.931031 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:07:37.964870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.964836 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.965338 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.965312 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.965768 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.965751 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.966220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.966203 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.967818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.967796 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.969516 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.969496 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.970145 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.970122 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.971111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.971092 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qkm7\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.971848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.968224 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.971978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.970965 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.972070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.966136 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.972222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.972202 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.972663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.972647 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.973374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.972553 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.995871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.995797 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:37.996061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:37.995920 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:38.029632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.029601 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:38.037448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.037422 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qkm7\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\") pod \"e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:38.159872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.159841 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:38.607205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.607175 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:07:38.607632 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:07:38.607609 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40a4fcb1_8f46_4bcd_8538_9c7734b42c73.slice/crio-6621b76bb896874c16390b510924fef134458c85baf84e3bd460f4d66adb5637 WatchSource:0}: Error finding container 6621b76bb896874c16390b510924fef134458c85baf84e3bd460f4d66adb5637: Status 404 returned error can't find the container with id 6621b76bb896874c16390b510924fef134458c85baf84e3bd460f4d66adb5637 Nov 03 18:07:38.610354 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.610322 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:38.610448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.610394 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:38.610448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.610428 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:07:38.969789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.969750 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerStarted","Data":"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968"} Nov 03 18:07:38.970259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:38.969846 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerStarted","Data":"6621b76bb896874c16390b510924fef134458c85baf84e3bd460f4d66adb5637"} Nov 03 18:07:39.100536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:39.100478 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podStartSLOduration=2.100459861 podStartE2EDuration="2.100459861s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:39.093814616 +0000 UTC m=+2038.922898219" watchObservedRunningTime="2026-11-03 18:07:39.100459861 +0000 UTC m=+2038.929543468" Nov 03 18:07:39.160445 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:39.160413 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:39.166533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:39.166510 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:39.978186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:39.978153 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:39.979226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:39.979203 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:07:40.901112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:40.901076 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:07:41.022275 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:41.022236 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:07:44.523120 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:07:44.523084 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:07:44.626464 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:44.626428 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 18:07:44.626750 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:44.626719 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" containerID="cri-o://552db5f262098ac1ec52a2153304d21d4f4c93f972f9e55effd2066f7d7dbc6f" gracePeriod=30 Nov 03 18:07:45.265643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.265605 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:07:45.265812 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.265669 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:07:45.832345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.832301 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:07:45.837107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.837082 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:45.903595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.900998 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-528e53-dockercfg-8m7tb\"" Nov 03 18:07:45.903804 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.903783 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-528e53\"" Nov 03 18:07:45.906606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.904519 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" containerID="cri-o://c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d" gracePeriod=25 Nov 03 18:07:45.940601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.932088 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:07:45.963776 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.963732 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:45.964098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:45.964071 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z679d\" (UniqueName: \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.078723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.042262 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" containerID="cri-o://f530de6bcda477c8a024cc4379c1f571fb4d624ff25fc27cfd6f1a87c9493029" gracePeriod=25 Nov 03 18:07:46.078723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.068674 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z679d\" (UniqueName: \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.078723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.069069 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.078723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.070068 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.149649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.145249 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z679d\" (UniqueName: \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\") pod \"payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.150808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.150786 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:46.580833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:46.580795 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:07:47.153536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:47.153498 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" event={"ID":"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1","Type":"ContainerStarted","Data":"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a"} Nov 03 18:07:47.153970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:47.153561 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" event={"ID":"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1","Type":"ContainerStarted","Data":"9ee53078ceaf2019b53baaf7f177889fb154fc4b06e70ac91074d9359bc3917b"} Nov 03 18:07:47.153970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:47.153608 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:07:47.273418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:47.273355 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" podStartSLOduration=2.273338539 podStartE2EDuration="2.273338539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:07:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:07:47.273000888 +0000 UTC m=+2047.102084493" watchObservedRunningTime="2026-11-03 18:07:47.273338539 +0000 UTC m=+2047.102422147" Nov 03 18:07:50.248005 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.247969 4980 generic.go:356] "Generic (PLEG): container finished" podID="33122dc2-1320-4916-82bd-ea2a19add58b" containerID="552db5f262098ac1ec52a2153304d21d4f4c93f972f9e55effd2066f7d7dbc6f" exitCode=0 Nov 03 18:07:50.248470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.248097 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" event={"ID":"33122dc2-1320-4916-82bd-ea2a19add58b","Type":"ContainerDied","Data":"552db5f262098ac1ec52a2153304d21d4f4c93f972f9e55effd2066f7d7dbc6f"} Nov 03 18:07:50.392353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.392314 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 18:07:50.443133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443108 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443144 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443174 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443200 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443229 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443308 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443336 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443386 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443398 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:07:50.443545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443439 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\") pod \"33122dc2-1320-4916-82bd-ea2a19add58b\" (UID: \"33122dc2-1320-4916-82bd-ea2a19add58b\") " Nov 03 18:07:50.443808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443645 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:07:50.443808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443647 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:07:50.443808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443764 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:07:50.443975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.443916 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:07:50.444603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.444556 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.444712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.444609 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.444712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.444624 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.444712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.444638 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.444712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.444652 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33122dc2-1320-4916-82bd-ea2a19add58b-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.445626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.445591 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:07:50.445818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.445798 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:07:50.445929 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.445856 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "kube-api-access-8hjtw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:07:50.445929 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.445901 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo" pod "33122dc2-1320-4916-82bd-ea2a19add58b" (UID: "33122dc2-1320-4916-82bd-ea2a19add58b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:07:50.546339 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.546308 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33122dc2-1320-4916-82bd-ea2a19add58b-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.546339 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.546337 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33122dc2-1320-4916-82bd-ea2a19add58b-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.546536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.546348 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjtw\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-kube-api-access-8hjtw\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.546536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.546370 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33122dc2-1320-4916-82bd-ea2a19add58b-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:07:50.665108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.665069 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:07:50.665305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:50.665084 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:07:51.259497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:51.259462 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" Nov 03 18:07:51.259497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:51.259484 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c" event={"ID":"33122dc2-1320-4916-82bd-ea2a19add58b","Type":"ContainerDied","Data":"4c8a29b94d96ffee674286fb7f1c2a7d55b6e59f49cd4a36818341b7ade30d5c"} Nov 03 18:07:51.260090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:51.259565 4980 scope.go:122] "RemoveContainer" containerID="552db5f262098ac1ec52a2153304d21d4f4c93f972f9e55effd2066f7d7dbc6f" Nov 03 18:07:51.481012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:51.480979 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 18:07:51.487039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:51.487015 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-dwx6c"] Nov 03 18:07:52.296477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:52.296442 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" path="/var/lib/kubelet/pods/33122dc2-1320-4916-82bd-ea2a19add58b/volumes" Nov 03 18:07:58.164585 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:07:58.164542 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:08:00.665521 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:00.665475 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:08:00.666039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:00.665515 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:08:06.285663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.285633 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:08:06.377828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.377793 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\" (UniqueName: \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\") pod \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " Nov 03 18:08:06.378423 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.378390 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\") pod \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\" (UID: \"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3\") " Nov 03 18:08:06.378955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.378852 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume" pod "aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" (UID: "aa3314f3-0313-40e8-b3ba-c6fe6c7732c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:08:06.380939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.380912 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:08:06.383196 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.383161 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l" pod "aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" (UID: "aa3314f3-0313-40e8-b3ba-c6fe6c7732c3"). InnerVolumeSpecName "kube-api-access-q8s8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:08:06.387117 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.387089 4980 generic.go:356] "Generic (PLEG): container finished" podID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerID="c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d" exitCode=0 Nov 03 18:08:06.387229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.387143 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" Nov 03 18:08:06.387518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.387314 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" event={"ID":"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3","Type":"ContainerDied","Data":"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d"} Nov 03 18:08:06.387518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.387365 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7" event={"ID":"aa3314f3-0313-40e8-b3ba-c6fe6c7732c3","Type":"ContainerDied","Data":"d1436da164c3cc07d5b1e793b685951fb32e0b25fd875dabf04f788648e0b81e"} Nov 03 18:08:06.387518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.387394 4980 scope.go:122] "RemoveContainer" containerID="c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d" Nov 03 18:08:06.391511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.391489 4980 generic.go:356] "Generic (PLEG): container finished" podID="2676795d-e35e-4387-93fc-85411a98d35f" containerID="f530de6bcda477c8a024cc4379c1f571fb4d624ff25fc27cfd6f1a87c9493029" exitCode=0 Nov 03 18:08:06.392044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.392017 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" event={"ID":"2676795d-e35e-4387-93fc-85411a98d35f","Type":"ContainerDied","Data":"f530de6bcda477c8a024cc4379c1f571fb4d624ff25fc27cfd6f1a87c9493029"} Nov 03 18:08:06.404807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.404789 4980 scope.go:122] "RemoveContainer" containerID="c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d" Nov 03 18:08:06.405125 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:08:06.405095 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d\": container with ID starting with c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d not found: ID does not exist" containerID="c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d" Nov 03 18:08:06.405203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.405137 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d"} err="failed to get container status \"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d\": rpc error: code = NotFound desc = could not find container \"c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d\": container with ID starting with c2f757e9bb556d4a127d62ba59274bd4143bef01d05b27cc2119eed88ba37f7d not found: ID does not exist" Nov 03 18:08:06.426461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.426439 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:08:06.481690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.481664 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\") pod \"2676795d-e35e-4387-93fc-85411a98d35f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " Nov 03 18:08:06.481951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.481931 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\" (UniqueName: \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\") pod \"2676795d-e35e-4387-93fc-85411a98d35f\" (UID: \"2676795d-e35e-4387-93fc-85411a98d35f\") " Nov 03 18:08:06.482066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.482044 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume" pod "2676795d-e35e-4387-93fc-85411a98d35f" (UID: "2676795d-e35e-4387-93fc-85411a98d35f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:08:06.482679 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.482658 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2676795d-e35e-4387-93fc-85411a98d35f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:08:06.482679 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.482680 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8s8l\" (UniqueName: \"kubernetes.io/projected/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3-kube-api-access-q8s8l\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:08:06.484208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.484190 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8" pod "2676795d-e35e-4387-93fc-85411a98d35f" (UID: "2676795d-e35e-4387-93fc-85411a98d35f"). InnerVolumeSpecName "kube-api-access-hdwc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:08:06.522531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.522454 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:08:06.528662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.528636 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-6ed3e95c-7c549d59b7-666q7"] Nov 03 18:08:06.584456 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:06.584425 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdwc8\" (UniqueName: \"kubernetes.io/projected/2676795d-e35e-4387-93fc-85411a98d35f-kube-api-access-hdwc8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:08:07.402094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:07.402017 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" Nov 03 18:08:07.402644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:07.402016 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f" event={"ID":"2676795d-e35e-4387-93fc-85411a98d35f","Type":"ContainerDied","Data":"086b45fe7ac919fcf3f16d87ec13c26a6f192cae7977e53e758221a2dd9b21b7"} Nov 03 18:08:07.402644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:07.402162 4980 scope.go:122] "RemoveContainer" containerID="f530de6bcda477c8a024cc4379c1f571fb4d624ff25fc27cfd6f1a87c9493029" Nov 03 18:08:07.579603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:07.579526 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:08:07.623920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:07.623886 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6ed3e95c-797d9d7975-q9p4f"] Nov 03 18:08:08.296545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:08.296510 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2676795d-e35e-4387-93fc-85411a98d35f" path="/var/lib/kubelet/pods/2676795d-e35e-4387-93fc-85411a98d35f/volumes" Nov 03 18:08:08.297252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:08.297226 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" path="/var/lib/kubelet/pods/aa3314f3-0313-40e8-b3ba-c6fe6c7732c3/volumes" Nov 03 18:08:14.533767 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:08:14.533730 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:08:21.091894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.091858 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:08:21.092659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092631 4980 cpu_manager.go:414] "containerMap: removing container" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" Nov 03 18:08:21.092659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092654 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" Nov 03 18:08:21.092821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092705 4980 cpu_manager.go:414] "containerMap: removing container" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" Nov 03 18:08:21.092821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092713 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" Nov 03 18:08:21.092821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092771 4980 cpu_manager.go:414] "containerMap: removing container" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" Nov 03 18:08:21.092821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092778 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" Nov 03 18:08:21.093073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.092981 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa3314f3-0313-40e8-b3ba-c6fe6c7732c3" containerName="payload-processing" Nov 03 18:08:21.093073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.093007 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="33122dc2-1320-4916-82bd-ea2a19add58b" containerName="istio-proxy" Nov 03 18:08:21.093073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.093021 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2676795d-e35e-4387-93fc-85411a98d35f" containerName="payload-pre-processing" Nov 03 18:08:21.106907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.106867 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.138974 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.138938 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:08:21.202597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.181989 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-640dc9-dockercfg-dwrjm\"" Nov 03 18:08:21.202597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.182093 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-640dc9\"" Nov 03 18:08:21.437647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.437276 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.437647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.437426 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhw9c\" (UniqueName: \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.538647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.538611 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.538824 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.538725 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhw9c\" (UniqueName: \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.539311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.539290 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.616011 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.615977 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhw9c\" (UniqueName: \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\") pod \"payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:21.726154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:21.726121 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:22.116108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:22.116077 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:08:22.117683 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:08:22.117558 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef07506_cbf6_4349_b89b_d363751b1d0c.slice/crio-a971f1459f75f48497a7f493e39b51142081f4335535c9d00d65e184e92d2633 WatchSource:0}: Error finding container a971f1459f75f48497a7f493e39b51142081f4335535c9d00d65e184e92d2633: Status 404 returned error can't find the container with id a971f1459f75f48497a7f493e39b51142081f4335535c9d00d65e184e92d2633 Nov 03 18:08:22.543472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:22.543438 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" event={"ID":"8ef07506-cbf6-4349-b89b-d363751b1d0c","Type":"ContainerStarted","Data":"ad283f9fb720666197d26a75aeae710725fc151497b2a2279a31054cbd836444"} Nov 03 18:08:22.543702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:22.543500 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" event={"ID":"8ef07506-cbf6-4349-b89b-d363751b1d0c","Type":"ContainerStarted","Data":"a971f1459f75f48497a7f493e39b51142081f4335535c9d00d65e184e92d2633"} Nov 03 18:08:22.543702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:22.543598 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:22.665522 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:22.665464 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" podStartSLOduration=1.66544932 podStartE2EDuration="1.66544932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:22.664811634 +0000 UTC m=+2082.493895236" watchObservedRunningTime="2026-11-03 18:08:22.66544932 +0000 UTC m=+2082.494532921" Nov 03 18:08:31.486493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.486459 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:08:31.491419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.491398 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.546365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.546328 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f5100c-openshift-default-dockercfg-hpszq\"" Nov 03 18:08:31.554894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.554848 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.554976 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555020 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555041 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555065 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555140 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555171 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszwj\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555341 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.555412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.555377 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.600791 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.600757 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:08:31.695601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.657642 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.716198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.716161 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.735678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.735643 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.742417 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.742329 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.743179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.743152 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszwj\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.743824 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.743801 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.744284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.744260 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.744928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.744903 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.745830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.745813 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.746334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.745662 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.746488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.744711 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.746608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.746163 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.746715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.727645 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.754045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.754005 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.760970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.760942 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.762686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.762661 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.806639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.806609 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszwj\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:31.806897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:31.806874 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\") pod \"e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:32.105319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.105224 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:32.551082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.551052 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:08:32.552023 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:08:32.551991 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod727ec5a8_b3d9_460d_afcc_342f989f21e3.slice/crio-b43a3e2508776401e6d8d3be1800cb852d8bfeff885e7c718144253ed04f8adf WatchSource:0}: Error finding container b43a3e2508776401e6d8d3be1800cb852d8bfeff885e7c718144253ed04f8adf: Status 404 returned error can't find the container with id b43a3e2508776401e6d8d3be1800cb852d8bfeff885e7c718144253ed04f8adf Nov 03 18:08:32.554864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.554834 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:32.554936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.554898 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:32.554978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.554939 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:32.654325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:32.654292 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerStarted","Data":"b43a3e2508776401e6d8d3be1800cb852d8bfeff885e7c718144253ed04f8adf"} Nov 03 18:08:33.554165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:33.554131 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:08:33.663769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:33.663722 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerStarted","Data":"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29"} Nov 03 18:08:33.802179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:33.802121 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" podStartSLOduration=2.802104164 podStartE2EDuration="2.802104164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:33.80093832 +0000 UTC m=+2093.630021921" watchObservedRunningTime="2026-11-03 18:08:33.802104164 +0000 UTC m=+2093.631187771" Nov 03 18:08:34.106072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:34.106038 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:34.112236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:34.112208 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:34.671424 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:34.671386 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:34.672515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:34.672493 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:08:40.952368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:40.952336 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:08:40.961326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:40.961298 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:08:40.961693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:40.961674 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.007113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.007085 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:08:41.011910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.011882 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:08:41.013450 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.013432 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:08:41.018694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.018676 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:08:41.073906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.073869 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f5100c\"" Nov 03 18:08:41.079854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.074193 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f5100c-dockercfg-xw8hl\"" Nov 03 18:08:41.080995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.080931 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hkhq\" (UniqueName: \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.081138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.081080 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.182756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.182713 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hkhq\" (UniqueName: \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.182756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.182758 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.183780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.183748 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.246340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.246255 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hkhq\" (UniqueName: \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\") pod \"payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.289763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.289728 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:41.702550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.702515 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:08:41.703531 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:08:41.703501 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9141a3f7_7938_4d70_a062_edec28a48a1c.slice/crio-2412080883de35d658b539c4fce248b7402fa721a340792e51cfc9e92aa2f4cf WatchSource:0}: Error finding container 2412080883de35d658b539c4fce248b7402fa721a340792e51cfc9e92aa2f4cf: Status 404 returned error can't find the container with id 2412080883de35d658b539c4fce248b7402fa721a340792e51cfc9e92aa2f4cf Nov 03 18:08:41.968606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.966802 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" event={"ID":"9141a3f7-7938-4d70-a062-edec28a48a1c","Type":"ContainerStarted","Data":"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf"} Nov 03 18:08:41.968606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.966907 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" event={"ID":"9141a3f7-7938-4d70-a062-edec28a48a1c","Type":"ContainerStarted","Data":"2412080883de35d658b539c4fce248b7402fa721a340792e51cfc9e92aa2f4cf"} Nov 03 18:08:41.973596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:41.971881 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:42.129664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:42.114772 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" podStartSLOduration=2.114753703 podStartE2EDuration="2.114753703s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:08:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:08:42.10273459 +0000 UTC m=+2101.931818200" watchObservedRunningTime="2026-11-03 18:08:42.114753703 +0000 UTC m=+2101.943837305" Nov 03 18:08:44.523373 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:08:44.523337 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:08:50.076337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.076301 4980 generic.go:356] "Generic (PLEG): container finished" podID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerID="6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194" exitCode=0 Nov 03 18:08:50.076846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.076378 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerDied","Data":"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194"} Nov 03 18:08:50.076961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.076942 4980 scope.go:122] "RemoveContainer" containerID="6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194" Nov 03 18:08:50.079291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.079256 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:50.079394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.079346 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:50.079438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:50.079395 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:08:51.090557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:51.090522 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerStarted","Data":"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a"} Nov 03 18:08:51.430812 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:51.430713 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:08:51.432378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:51.432349 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:51.432514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:51.432399 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:52.431008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:52.430968 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:52.431512 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:52.431037 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:52.510838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:52.510799 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:08:53.431397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:53.431358 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:53.431844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:53.431439 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:54.000654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:54.000623 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:08:54.433602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:54.433486 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:54.434252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:54.434215 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:55.431443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:55.431405 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:55.431679 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:55.431473 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:56.431537 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:56.431500 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" start-of-body= Nov 03 18:08:56.432115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:56.431589 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.131:15021/healthz/ready\": dial tcp 10.132.0.131:15021: connect: connection refused" Nov 03 18:08:57.436592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:57.436545 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:08:57.437473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:08:57.437452 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:09:14.522797 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:09:14.522759 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:09:44.522301 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:09:44.522257 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:09:57.414233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:09:57.414197 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:09:57.531369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:09:57.531313 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:09:59.623763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:09:59.623717 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:09:59.624153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:09:59.623794 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:10:02.414566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.414522 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" podUID="3e8378a5-4ac6-4726-81ae-070a3937d984" containerName="payload-processing" containerID="cri-o://4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e" gracePeriod=25 Nov 03 18:10:02.532263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.532218 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" podUID="6432b682-dbb3-493b-b52f-374822e176c7" containerName="payload-pre-processing" containerID="cri-o://ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7" gracePeriod=25 Nov 03 18:10:02.826327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.826289 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:10:02.931788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.931751 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:10:02.979806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.979773 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\" (UniqueName: \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\") pod \"3e8378a5-4ac6-4726-81ae-070a3937d984\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " Nov 03 18:10:02.980041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.979921 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\") pod \"3e8378a5-4ac6-4726-81ae-070a3937d984\" (UID: \"3e8378a5-4ac6-4726-81ae-070a3937d984\") " Nov 03 18:10:02.980364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.980341 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume" pod "3e8378a5-4ac6-4726-81ae-070a3937d984" (UID: "3e8378a5-4ac6-4726-81ae-070a3937d984"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:10:02.980719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.980703 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e8378a5-4ac6-4726-81ae-070a3937d984-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:02.982441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:02.982415 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h" pod "3e8378a5-4ac6-4726-81ae-070a3937d984" (UID: "3e8378a5-4ac6-4726-81ae-070a3937d984"). InnerVolumeSpecName "kube-api-access-s9h2h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:10:03.082286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.082241 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\" (UniqueName: \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\") pod \"6432b682-dbb3-493b-b52f-374822e176c7\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.082443 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\") pod \"6432b682-dbb3-493b-b52f-374822e176c7\" (UID: \"6432b682-dbb3-493b-b52f-374822e176c7\") " Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.082791 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume" pod "6432b682-dbb3-493b-b52f-374822e176c7" (UID: "6432b682-dbb3-493b-b52f-374822e176c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.083032 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6432b682-dbb3-493b-b52f-374822e176c7-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.083052 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9h2h\" (UniqueName: \"kubernetes.io/projected/3e8378a5-4ac6-4726-81ae-070a3937d984-kube-api-access-s9h2h\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.085100 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd" pod "6432b682-dbb3-493b-b52f-374822e176c7" (UID: "6432b682-dbb3-493b-b52f-374822e176c7"). InnerVolumeSpecName "kube-api-access-gn5pd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.184897 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn5pd\" (UniqueName: \"kubernetes.io/projected/6432b682-dbb3-493b-b52f-374822e176c7-kube-api-access-gn5pd\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.201410 4980 generic.go:356] "Generic (PLEG): container finished" podID="3e8378a5-4ac6-4726-81ae-070a3937d984" containerID="4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e" exitCode=0 Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.201459 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.201496 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" event={"ID":"3e8378a5-4ac6-4726-81ae-070a3937d984","Type":"ContainerDied","Data":"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e"} Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.201547 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj" event={"ID":"3e8378a5-4ac6-4726-81ae-070a3937d984","Type":"ContainerDied","Data":"528cb9eb6d9b88cb61fe1420cd9385e5a7fb12df0e1832f2255324eb4751b89c"} Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.201614 4980 scope.go:122] "RemoveContainer" containerID="4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.203254 4980 generic.go:356] "Generic (PLEG): container finished" podID="6432b682-dbb3-493b-b52f-374822e176c7" containerID="ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7" exitCode=0 Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.203357 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.203419 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" event={"ID":"6432b682-dbb3-493b-b52f-374822e176c7","Type":"ContainerDied","Data":"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7"} Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.203463 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s" event={"ID":"6432b682-dbb3-493b-b52f-374822e176c7","Type":"ContainerDied","Data":"a885dc74fa852f8c6164bcb690c7ef933e3e9d03be389257fae9f7187511186b"} Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.217730 4980 scope.go:122] "RemoveContainer" containerID="4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:03.218086 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e\": container with ID starting with 4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e not found: ID does not exist" containerID="4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.218112 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e"} err="failed to get container status \"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e\": rpc error: code = NotFound desc = could not find container \"4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e\": container with ID starting with 4c5a8d77dd9059011c77fdfc0da9840074751a71b69401a702fe199adebade1e not found: ID does not exist" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.218131 4980 scope.go:122] "RemoveContainer" containerID="ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.311606 4980 scope.go:122] "RemoveContainer" containerID="ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:03.312012 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7\": container with ID starting with ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7 not found: ID does not exist" containerID="ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.312039 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7"} err="failed to get container status \"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7\": rpc error: code = NotFound desc = could not find container \"ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7\": container with ID starting with ba9baa8e0e7ae6c4c6c474a1198cce2f9c090cea3e57dd8a0b5fa945bd1700e7 not found: ID does not exist" Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.371759 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.378277 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-d9a300-7f674658c5-gxmmj"] Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.490189 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:10:03.728829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:03.495340 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d9a300-65975cd675-hfd2s"] Nov 03 18:10:04.297139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:04.297098 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e8378a5-4ac6-4726-81ae-070a3937d984" path="/var/lib/kubelet/pods/3e8378a5-4ac6-4726-81ae-070a3937d984/volumes" Nov 03 18:10:04.297731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:04.297708 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6432b682-dbb3-493b-b52f-374822e176c7" path="/var/lib/kubelet/pods/6432b682-dbb3-493b-b52f-374822e176c7/volumes" Nov 03 18:10:04.487062 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:04.487020 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:04.487273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:04.487088 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:05.222430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.222368 4980 generic.go:356] "Generic (PLEG): container finished" podID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerID="df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408" exitCode=0 Nov 03 18:10:05.222979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.222455 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerDied","Data":"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408"} Nov 03 18:10:05.223103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.223080 4980 scope.go:122] "RemoveContainer" containerID="df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408" Nov 03 18:10:05.225256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.225225 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:05.225364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.225318 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:05.225364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:05.225355 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:06.235471 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:06.235440 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerStarted","Data":"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f"} Nov 03 18:10:06.901987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:06.901955 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:07.007533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:06.902108 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:07.007533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:06.902160 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:07.247930 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.247897 4980 generic.go:356] "Generic (PLEG): container finished" podID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerID="b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968" exitCode=0 Nov 03 18:10:07.248314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.247970 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerDied","Data":"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968"} Nov 03 18:10:07.248754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.248738 4980 scope.go:122] "RemoveContainer" containerID="b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968" Nov 03 18:10:07.250427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.250401 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:07.250503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.250468 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:07.250503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.250498 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:07.902391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.902294 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:07.902391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:07.902357 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:08.275924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.269227 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerStarted","Data":"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49"} Nov 03 18:10:08.287597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.286485 4980 generic.go:356] "Generic (PLEG): container finished" podID="f36b392c-adef-417f-883f-ee3d30774714" containerID="3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111" exitCode=0 Nov 03 18:10:08.287597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.286657 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerDied","Data":"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111"} Nov 03 18:10:08.287597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.287208 4980 scope.go:122] "RemoveContainer" containerID="3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111" Nov 03 18:10:08.311330 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.301996 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:08.311330 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.302099 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:08.311330 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.302145 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:10:08.913087 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.913045 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:08.913378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:08.913347 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:09.174913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.174825 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:10:09.197754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.197713 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:09.199247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.199212 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:09.347245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.347203 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerStarted","Data":"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d"} Nov 03 18:10:09.606993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.606959 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:10:09.607381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.607358 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:09.607591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.607542 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:09.905435 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.905336 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:09.905750 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.905719 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:09.983599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:09.983545 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:10:10.163540 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.163028 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:10.163540 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.163092 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:10.603028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.602992 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:10.603657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.603626 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:10.919090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.918993 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:10.919350 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:10.919321 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:11.170652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.169832 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:11.170652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.169909 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:11.603196 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.603158 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:11.604331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.603812 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:11.904045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.903949 4980 patch_prober.go:29] interesting pod/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" start-of-body= Nov 03 18:10:11.904323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:11.904290 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.137:15021/healthz/ready\": dial tcp 10.132.0.137:15021: connect: connection refused" Nov 03 18:10:12.185655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.172006 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:12.185655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.172094 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:12.603598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.602962 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:12.603598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.603044 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:12.917975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.917888 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:12.920237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.920209 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:12.927094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:12.927062 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:13.174644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:13.174524 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:13.175095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:13.175061 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:13.591893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:13.591859 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:10:13.603977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:13.603940 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:13.604591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:13.604538 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:14.167066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:14.167028 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" start-of-body= Nov 03 18:10:14.167408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:14.167360 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.139:15021/healthz/ready\": dial tcp 10.132.0.139:15021: connect: connection refused" Nov 03 18:10:14.523058 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:14.523020 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:10:14.602472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:14.602429 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" start-of-body= Nov 03 18:10:14.602680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:14.602496 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.132:15021/healthz/ready\": dial tcp 10.132.0.132:15021: connect: connection refused" Nov 03 18:10:15.176608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:15.175849 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:10:15.185854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:15.185802 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:10:15.680746 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:15.680704 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:10:15.682257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:15.682228 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:10:31.904661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:31.904194 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:10:36.923783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:36.923723 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" podUID="07107523-9d65-4545-a3f6-26474bf83639" containerName="payload-pre-processing" containerID="cri-o://427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61" gracePeriod=25 Nov 03 18:10:37.308515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.308482 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:10:37.388910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.388880 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\") pod \"07107523-9d65-4545-a3f6-26474bf83639\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " Nov 03 18:10:37.389161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.389141 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\" (UniqueName: \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\") pod \"07107523-9d65-4545-a3f6-26474bf83639\" (UID: \"07107523-9d65-4545-a3f6-26474bf83639\") " Nov 03 18:10:37.389298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.389279 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume" pod "07107523-9d65-4545-a3f6-26474bf83639" (UID: "07107523-9d65-4545-a3f6-26474bf83639"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:10:37.389927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.389901 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07107523-9d65-4545-a3f6-26474bf83639-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:37.391398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.391370 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t" pod "07107523-9d65-4545-a3f6-26474bf83639" (UID: "07107523-9d65-4545-a3f6-26474bf83639"). InnerVolumeSpecName "kube-api-access-fvl4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:10:37.491188 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.491097 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvl4t\" (UniqueName: \"kubernetes.io/projected/07107523-9d65-4545-a3f6-26474bf83639-kube-api-access-fvl4t\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:37.712012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.711977 4980 generic.go:356] "Generic (PLEG): container finished" podID="07107523-9d65-4545-a3f6-26474bf83639" containerID="427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61" exitCode=0 Nov 03 18:10:37.712231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.712040 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" Nov 03 18:10:37.712231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.712066 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" event={"ID":"07107523-9d65-4545-a3f6-26474bf83639","Type":"ContainerDied","Data":"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61"} Nov 03 18:10:37.712231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.712123 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk" event={"ID":"07107523-9d65-4545-a3f6-26474bf83639","Type":"ContainerDied","Data":"68b2a72e70c68760e0e4513bdd92fd7f6d7f7bef1f9419df313f231e39445bf3"} Nov 03 18:10:37.712231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.712146 4980 scope.go:122] "RemoveContainer" containerID="427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61" Nov 03 18:10:37.724660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.724634 4980 scope.go:122] "RemoveContainer" containerID="427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61" Nov 03 18:10:37.724981 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:37.724962 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61\": container with ID starting with 427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61 not found: ID does not exist" containerID="427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61" Nov 03 18:10:37.725038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.724992 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61"} err="failed to get container status \"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61\": rpc error: code = NotFound desc = could not find container \"427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61\": container with ID starting with 427be61bcaad6e74884bd3a88e644e51f2cf4b482bc18a1cdf626e199fedee61 not found: ID does not exist" Nov 03 18:10:37.882797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.882713 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:10:37.888118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:37.888090 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-c17c6a-d8ff5f69f-j7xsk"] Nov 03 18:10:38.296268 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:38.296235 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07107523-9d65-4545-a3f6-26474bf83639" path="/var/lib/kubelet/pods/07107523-9d65-4545-a3f6-26474bf83639/volumes" Nov 03 18:10:44.521676 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:44.521643 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:10:47.039499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:47.039465 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:10:47.039960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:47.039753 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" containerID="cri-o://855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f" gracePeriod=30 Nov 03 18:10:52.431830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.431793 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:52.521358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521331 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521454 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521472 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521533 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521593 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521611 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:10:52.521771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521644 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521726 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.521771 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521768 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.522728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521790 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:10:52.522728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521803 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\") pod \"07531d91-a805-4751-aa5e-5aca33cbccdc\" (UID: \"07531d91-a805-4751-aa5e-5aca33cbccdc\") " Nov 03 18:10:52.522728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521938 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:10:52.522728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.521977 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:10:52.522728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522182 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:10:52.523026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522771 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.523026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522795 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.523026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522811 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.523026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522827 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07531d91-a805-4751-aa5e-5aca33cbccdc-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.523026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.522844 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.524108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.524082 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:10:52.524236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.524183 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:10:52.524507 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.524482 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:10:52.524625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.524554 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc" pod "07531d91-a805-4751-aa5e-5aca33cbccdc" (UID: "07531d91-a805-4751-aa5e-5aca33cbccdc"). InnerVolumeSpecName "kube-api-access-l7vmc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:10:52.624374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.624342 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.624374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.624373 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l7vmc\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-kube-api-access-l7vmc\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.624584 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.624384 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.624584 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.624393 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07531d91-a805-4751-aa5e-5aca33cbccdc-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:10:52.872921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.872888 4980 generic.go:356] "Generic (PLEG): container finished" podID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerID="855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f" exitCode=0 Nov 03 18:10:52.873095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.872935 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" Nov 03 18:10:52.873095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.872973 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerDied","Data":"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f"} Nov 03 18:10:52.873095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.873033 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx" event={"ID":"07531d91-a805-4751-aa5e-5aca33cbccdc","Type":"ContainerDied","Data":"812ebb4c10d37c9ad75f5cfbdf31821b13e79398922d345bd8671f914f54855d"} Nov 03 18:10:52.873095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.873060 4980 scope.go:122] "RemoveContainer" containerID="855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f" Nov 03 18:10:52.885261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.885241 4980 scope.go:122] "RemoveContainer" containerID="df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408" Nov 03 18:10:52.896102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.896078 4980 scope.go:122] "RemoveContainer" containerID="855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f" Nov 03 18:10:52.896413 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:52.896393 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f\": container with ID starting with 855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f not found: ID does not exist" containerID="855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f" Nov 03 18:10:52.896482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.896422 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f"} err="failed to get container status \"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f\": rpc error: code = NotFound desc = could not find container \"855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f\": container with ID starting with 855e7e77a43357e0dea28ae27c8b2befbbe0cf5e4e8d5cf60b7a1aa3550c934f not found: ID does not exist" Nov 03 18:10:52.896482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.896442 4980 scope.go:122] "RemoveContainer" containerID="df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408" Nov 03 18:10:52.896758 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:10:52.896739 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408\": container with ID starting with df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408 not found: ID does not exist" containerID="df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408" Nov 03 18:10:52.896806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:52.896764 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408"} err="failed to get container status \"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408\": rpc error: code = NotFound desc = could not find container \"df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408\": container with ID starting with df5bb7ac3ac6c69f8676f241c68cdf8d13555d35f4498711a844be4893b8d408 not found: ID does not exist" Nov 03 18:10:53.044564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:53.044523 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:10:53.094787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:53.094760 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c17c6a-openshift-default-55449dfcb5-wmktx"] Nov 03 18:10:54.296053 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:54.296017 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" path="/var/lib/kubelet/pods/07531d91-a805-4751-aa5e-5aca33cbccdc/volumes" Nov 03 18:10:54.985874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:54.985842 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:10:59.995595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:10:59.994647 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerName="payload-pre-processing" containerID="cri-o://a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821" gracePeriod=25 Nov 03 18:11:00.333788 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:11:00.329613 4980 logging.go:55] [core] [Channel #3167 SubChannel #3168]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.122:50052", ServerName: "10.132.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.122:50052: connect: connection refused" Nov 03 18:11:00.744332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.744301 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:11:00.875556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.872377 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\") pod \"ea5267ea-f827-4810-a5bb-93032a869f8e\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " Nov 03 18:11:00.875935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.875908 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\" (UniqueName: \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\") pod \"ea5267ea-f827-4810-a5bb-93032a869f8e\" (UID: \"ea5267ea-f827-4810-a5bb-93032a869f8e\") " Nov 03 18:11:00.876430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.876371 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume" pod "ea5267ea-f827-4810-a5bb-93032a869f8e" (UID: "ea5267ea-f827-4810-a5bb-93032a869f8e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:00.882306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.882277 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea5267ea-f827-4810-a5bb-93032a869f8e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:00.885789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.885755 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl" pod "ea5267ea-f827-4810-a5bb-93032a869f8e" (UID: "ea5267ea-f827-4810-a5bb-93032a869f8e"). InnerVolumeSpecName "kube-api-access-wcxpl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:00.940940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.940902 4980 generic.go:356] "Generic (PLEG): container finished" podID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerID="a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821" exitCode=0 Nov 03 18:11:00.941137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.940968 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" Nov 03 18:11:00.941137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.941042 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" event={"ID":"ea5267ea-f827-4810-a5bb-93032a869f8e","Type":"ContainerDied","Data":"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821"} Nov 03 18:11:00.941137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.941091 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" event={"ID":"ea5267ea-f827-4810-a5bb-93032a869f8e","Type":"ContainerDied","Data":"23fa35c8c562c4d65da707bf63d99ed3de3f95dd77d408804c8f888905b69c9a"} Nov 03 18:11:00.941137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.941118 4980 scope.go:122] "RemoveContainer" containerID="a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821" Nov 03 18:11:00.952418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.952284 4980 scope.go:122] "RemoveContainer" containerID="a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821" Nov 03 18:11:00.952693 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:00.952602 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821\": container with ID starting with a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821 not found: ID does not exist" containerID="a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821" Nov 03 18:11:00.952693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.952634 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821"} err="failed to get container status \"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821\": rpc error: code = NotFound desc = could not find container \"a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821\": container with ID starting with a91a3710b219300d9af1e53eecbc7aa5443138f6ca35c3082ca246331f831821 not found: ID does not exist" Nov 03 18:11:00.984207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:00.984181 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcxpl\" (UniqueName: \"kubernetes.io/projected/ea5267ea-f827-4810-a5bb-93032a869f8e-kube-api-access-wcxpl\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:01.110669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:01.110631 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:11:01.123721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:01.123681 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4"] Nov 03 18:11:01.328721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:01.328670 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-8dfc64-55576c78f9-wfbb4" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.122:50052\" within 1s: context deadline exceeded" Nov 03 18:11:02.218173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:02.218137 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:11:02.296817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:02.296789 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" path="/var/lib/kubelet/pods/ea5267ea-f827-4810-a5bb-93032a869f8e/volumes" Nov 03 18:11:02.332911 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:02.332871 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:11:07.219291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.219245 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" podUID="f047702e-2c8d-4ca3-9526-7067055dc854" containerName="payload-processing" containerID="cri-o://3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02" gracePeriod=25 Nov 03 18:11:07.333353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.333315 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" podUID="67767ae8-7eab-4610-968f-2511f3569102" containerName="payload-pre-processing" containerID="cri-o://cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01" gracePeriod=25 Nov 03 18:11:07.518288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.518259 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:11:07.616194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.616166 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:11:07.616724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.616705 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\") pod \"f047702e-2c8d-4ca3-9526-7067055dc854\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " Nov 03 18:11:07.616977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.616916 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\" (UniqueName: \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\") pod \"f047702e-2c8d-4ca3-9526-7067055dc854\" (UID: \"f047702e-2c8d-4ca3-9526-7067055dc854\") " Nov 03 18:11:07.617120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.617094 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume" pod "f047702e-2c8d-4ca3-9526-7067055dc854" (UID: "f047702e-2c8d-4ca3-9526-7067055dc854"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:07.617536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.617515 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f047702e-2c8d-4ca3-9526-7067055dc854-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:07.619043 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.619021 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5" pod "f047702e-2c8d-4ca3-9526-7067055dc854" (UID: "f047702e-2c8d-4ca3-9526-7067055dc854"). InnerVolumeSpecName "kube-api-access-dvlp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:07.718690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.718653 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\") pod \"67767ae8-7eab-4610-968f-2511f3569102\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " Nov 03 18:11:07.718690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.718694 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\" (UniqueName: \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\") pod \"67767ae8-7eab-4610-968f-2511f3569102\" (UID: \"67767ae8-7eab-4610-968f-2511f3569102\") " Nov 03 18:11:07.719014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.718987 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume" pod "67767ae8-7eab-4610-968f-2511f3569102" (UID: "67767ae8-7eab-4610-968f-2511f3569102"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:07.719438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.719416 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvlp5\" (UniqueName: \"kubernetes.io/projected/f047702e-2c8d-4ca3-9526-7067055dc854-kube-api-access-dvlp5\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:07.719525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.719439 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/67767ae8-7eab-4610-968f-2511f3569102-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:07.720915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.720894 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4" pod "67767ae8-7eab-4610-968f-2511f3569102" (UID: "67767ae8-7eab-4610-968f-2511f3569102"). InnerVolumeSpecName "kube-api-access-kr7d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:07.820438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:07.820360 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr7d4\" (UniqueName: \"kubernetes.io/projected/67767ae8-7eab-4610-968f-2511f3569102-kube-api-access-kr7d4\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:08.017514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.017483 4980 generic.go:356] "Generic (PLEG): container finished" podID="67767ae8-7eab-4610-968f-2511f3569102" containerID="cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01" exitCode=0 Nov 03 18:11:08.017703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.017534 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" Nov 03 18:11:08.017703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.017603 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" event={"ID":"67767ae8-7eab-4610-968f-2511f3569102","Type":"ContainerDied","Data":"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01"} Nov 03 18:11:08.017703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.017650 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn" event={"ID":"67767ae8-7eab-4610-968f-2511f3569102","Type":"ContainerDied","Data":"d98dc0c84f425a83d435f9f6aabfc3e267274f94f839184085b350e21696fb84"} Nov 03 18:11:08.017703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.017671 4980 scope.go:122] "RemoveContainer" containerID="cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01" Nov 03 18:11:08.019263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.019241 4980 generic.go:356] "Generic (PLEG): container finished" podID="f047702e-2c8d-4ca3-9526-7067055dc854" containerID="3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02" exitCode=0 Nov 03 18:11:08.019358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.019276 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" Nov 03 18:11:08.019358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.019325 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" event={"ID":"f047702e-2c8d-4ca3-9526-7067055dc854","Type":"ContainerDied","Data":"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02"} Nov 03 18:11:08.019451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.019366 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68" event={"ID":"f047702e-2c8d-4ca3-9526-7067055dc854","Type":"ContainerDied","Data":"36f7f5cdabf319387be8f5eca56a279567c73981e61d51f7716ecee5545d8f5b"} Nov 03 18:11:08.030380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.030362 4980 scope.go:122] "RemoveContainer" containerID="cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01" Nov 03 18:11:08.030659 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:08.030642 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01\": container with ID starting with cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01 not found: ID does not exist" containerID="cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01" Nov 03 18:11:08.030719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.030666 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01"} err="failed to get container status \"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01\": rpc error: code = NotFound desc = could not find container \"cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01\": container with ID starting with cdbde1d27aec929c3882a2c2811e76ea4a55f4fa7d3b6e39a2ebdf1b28ae9b01 not found: ID does not exist" Nov 03 18:11:08.030719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.030684 4980 scope.go:122] "RemoveContainer" containerID="3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02" Nov 03 18:11:08.043650 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.043628 4980 scope.go:122] "RemoveContainer" containerID="3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02" Nov 03 18:11:08.043953 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:08.043927 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02\": container with ID starting with 3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02 not found: ID does not exist" containerID="3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02" Nov 03 18:11:08.044014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.043965 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02"} err="failed to get container status \"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02\": rpc error: code = NotFound desc = could not find container \"3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02\": container with ID starting with 3fba5989cf2517ef56bc5daff84d155511c9a3da59b845cdba75d4311f74cd02 not found: ID does not exist" Nov 03 18:11:08.189492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.189464 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:11:08.239910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.239869 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-e75517-7b46c55797-t9d68"] Nov 03 18:11:08.296989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.296959 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f047702e-2c8d-4ca3-9526-7067055dc854" path="/var/lib/kubelet/pods/f047702e-2c8d-4ca3-9526-7067055dc854/volumes" Nov 03 18:11:08.306245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.306218 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:11:08.311868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:08.311838 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-e75517-d699f8cff-xl5fn"] Nov 03 18:11:10.296551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:10.296514 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67767ae8-7eab-4610-968f-2511f3569102" path="/var/lib/kubelet/pods/67767ae8-7eab-4610-968f-2511f3569102/volumes" Nov 03 18:11:10.749528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:10.749492 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:11:10.749819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:10.749717 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" podUID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" containerName="maas-api" containerID="cri-o://00ea6009572e958a1885efc070f9f9688794c6d6e74ff2025c862738c836e034" gracePeriod=30 Nov 03 18:11:11.111649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.111611 4980 generic.go:356] "Generic (PLEG): container finished" podID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" containerID="00ea6009572e958a1885efc070f9f9688794c6d6e74ff2025c862738c836e034" exitCode=0 Nov 03 18:11:11.112180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.112149 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" event={"ID":"0a0fadc6-b470-4f2a-ae09-889668bfcdc3","Type":"ContainerDied","Data":"00ea6009572e958a1885efc070f9f9688794c6d6e74ff2025c862738c836e034"} Nov 03 18:11:11.112359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.112343 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" event={"ID":"0a0fadc6-b470-4f2a-ae09-889668bfcdc3","Type":"ContainerDied","Data":"ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69"} Nov 03 18:11:11.112528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.112507 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca3ce3034716386ad63398d5b1b14ef255f4274dccf65dd1a3adaaf24198da69" Nov 03 18:11:11.127344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.127320 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:11:11.180238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.180197 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\") pod \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " Nov 03 18:11:11.182299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.182259 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\" (UniqueName: \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\") pod \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " Nov 03 18:11:11.182991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.182963 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") pod \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " Nov 03 18:11:11.183494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.183474 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\") pod \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\" (UID: \"0a0fadc6-b470-4f2a-ae09-889668bfcdc3\") " Nov 03 18:11:11.188845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.188816 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca" pod "0a0fadc6-b470-4f2a-ae09-889668bfcdc3" (UID: "0a0fadc6-b470-4f2a-ae09-889668bfcdc3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:11.192454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.192410 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls" pod "0a0fadc6-b470-4f2a-ae09-889668bfcdc3" (UID: "0a0fadc6-b470-4f2a-ae09-889668bfcdc3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:11:11.226897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.220221 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs" pod "0a0fadc6-b470-4f2a-ae09-889668bfcdc3" (UID: "0a0fadc6-b470-4f2a-ae09-889668bfcdc3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:11:11.227795 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.227756 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b" pod "0a0fadc6-b470-4f2a-ae09-889668bfcdc3" (UID: "0a0fadc6-b470-4f2a-ae09-889668bfcdc3"). InnerVolumeSpecName "kube-api-access-c5l6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:11.290834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.290800 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5l6b\" (UniqueName: \"kubernetes.io/projected/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-kube-api-access-c5l6b\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:11.301130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.301083 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:11.301801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.301780 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:11.301988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:11.301960 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a0fadc6-b470-4f2a-ae09-889668bfcdc3-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:12.108242 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.108204 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:11:12.108429 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.108403 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" podUID="eb441d60-e513-41d0-b889-17964b4d2ac3" containerName="maas-api" containerID="cri-o://760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9" gracePeriod=30 Nov 03 18:11:12.121133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.121097 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv" Nov 03 18:11:12.362359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.362230 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:11:12.363008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.362988 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w6-ad914d-6cf87585c-l5lxv"] Nov 03 18:11:12.630648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.621450 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:11:12.702470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.702434 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") pod \"eb441d60-e513-41d0-b889-17964b4d2ac3\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " Nov 03 18:11:12.702659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.702506 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\") pod \"eb441d60-e513-41d0-b889-17964b4d2ac3\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " Nov 03 18:11:12.702659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.702563 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\" (UniqueName: \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\") pod \"eb441d60-e513-41d0-b889-17964b4d2ac3\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " Nov 03 18:11:12.702754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.702659 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\") pod \"eb441d60-e513-41d0-b889-17964b4d2ac3\" (UID: \"eb441d60-e513-41d0-b889-17964b4d2ac3\") " Nov 03 18:11:12.702967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.702942 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca" pod "eb441d60-e513-41d0-b889-17964b4d2ac3" (UID: "eb441d60-e513-41d0-b889-17964b4d2ac3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:12.703309 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.703246 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb441d60-e513-41d0-b889-17964b4d2ac3-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:12.704876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.704854 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls" pod "eb441d60-e513-41d0-b889-17964b4d2ac3" (UID: "eb441d60-e513-41d0-b889-17964b4d2ac3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:11:12.704967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.704890 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs" pod "eb441d60-e513-41d0-b889-17964b4d2ac3" (UID: "eb441d60-e513-41d0-b889-17964b4d2ac3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:11:12.704967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.704893 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt" pod "eb441d60-e513-41d0-b889-17964b4d2ac3" (UID: "eb441d60-e513-41d0-b889-17964b4d2ac3"). InnerVolumeSpecName "kube-api-access-gzvqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:12.804997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.804960 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzvqt\" (UniqueName: \"kubernetes.io/projected/eb441d60-e513-41d0-b889-17964b4d2ac3-kube-api-access-gzvqt\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:12.804997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.804995 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:12.804997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:12.805004 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb441d60-e513-41d0-b889-17964b4d2ac3-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:13.157422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.157386 4980 generic.go:356] "Generic (PLEG): container finished" podID="eb441d60-e513-41d0-b889-17964b4d2ac3" containerID="760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9" exitCode=0 Nov 03 18:11:13.157826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.157798 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" event={"ID":"eb441d60-e513-41d0-b889-17964b4d2ac3","Type":"ContainerDied","Data":"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9"} Nov 03 18:11:13.161702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.161661 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" event={"ID":"eb441d60-e513-41d0-b889-17964b4d2ac3","Type":"ContainerDied","Data":"9172cf0b1dfad537e7b12e7a2f7e74ef9c2cc1c1bef45538750770efb6d4ba9b"} Nov 03 18:11:13.162071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.161962 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49" Nov 03 18:11:13.171603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.161998 4980 scope.go:122] "RemoveContainer" containerID="760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9" Nov 03 18:11:13.212008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.211553 4980 scope.go:122] "RemoveContainer" containerID="760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9" Nov 03 18:11:13.212544 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:13.212442 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9\": container with ID starting with 760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9 not found: ID does not exist" containerID="760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9" Nov 03 18:11:13.212544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.212486 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9"} err="failed to get container status \"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9\": rpc error: code = NotFound desc = could not find container \"760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9\": container with ID starting with 760cbf36716d897e89776b7cabe29b61d0e4c6fdaa80414e4eb2915cb345afd9 not found: ID does not exist" Nov 03 18:11:13.350933 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.350896 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:11:13.365227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.365196 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-d9a300-58747f8cf7-7nn49"] Nov 03 18:11:13.958353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.958313 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:11:13.958609 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:13.958557 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" containerID="cri-o://a1210767430e43378e0878c687aa1b773cd459dedefc8e0bbb1427e1a4593128" gracePeriod=30 Nov 03 18:11:14.295361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:14.295285 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" path="/var/lib/kubelet/pods/0a0fadc6-b470-4f2a-ae09-889668bfcdc3/volumes" Nov 03 18:11:14.295807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:14.295792 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb441d60-e513-41d0-b889-17964b4d2ac3" path="/var/lib/kubelet/pods/eb441d60-e513-41d0-b889-17964b4d2ac3/volumes" Nov 03 18:11:14.521771 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:14.521735 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:11:19.205085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.205053 4980 generic.go:356] "Generic (PLEG): container finished" podID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerID="a1210767430e43378e0878c687aa1b773cd459dedefc8e0bbb1427e1a4593128" exitCode=0 Nov 03 18:11:19.205608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.205124 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" event={"ID":"7e7909f6-bccc-462b-9c42-fc120e4dc653","Type":"ContainerDied","Data":"a1210767430e43378e0878c687aa1b773cd459dedefc8e0bbb1427e1a4593128"} Nov 03 18:11:19.282013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.281989 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:11:19.320745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.320703 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.320941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.320853 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.320941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.320899 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.320944 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:19.321075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.320958 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321010 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321114 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321130 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321167 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321224 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321255 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\") pod \"7e7909f6-bccc-462b-9c42-fc120e4dc653\" (UID: \"7e7909f6-bccc-462b-9c42-fc120e4dc653\") " Nov 03 18:11:19.321315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321272 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:19.321662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321399 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:19.321662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.321549 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:19.322044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.322021 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.322140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.322048 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.322140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.322102 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7e7909f6-bccc-462b-9c42-fc120e4dc653-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.322140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.322118 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.322313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.322166 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.323712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.323689 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:11:19.323991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.323965 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:19.323991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.323972 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:19.324127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.324011 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm" pod "7e7909f6-bccc-462b-9c42-fc120e4dc653" (UID: "7e7909f6-bccc-462b-9c42-fc120e4dc653"). InnerVolumeSpecName "kube-api-access-5zwgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:19.423899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.423862 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.423899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.423887 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zwgm\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-kube-api-access-5zwgm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.424099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.423904 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:19.424099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:19.423918 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7e7909f6-bccc-462b-9c42-fc120e4dc653-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:20.214716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:20.214687 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" Nov 03 18:11:20.215805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:20.214726 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4" event={"ID":"7e7909f6-bccc-462b-9c42-fc120e4dc653","Type":"ContainerDied","Data":"e3503dd7c3260dc5715a82a8c77d67444eb0232b8683848325d2d4d13730dcbe"} Nov 03 18:11:20.215805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:20.214788 4980 scope.go:122] "RemoveContainer" containerID="a1210767430e43378e0878c687aa1b773cd459dedefc8e0bbb1427e1a4593128" Nov 03 18:11:20.435271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:20.435232 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:11:20.484865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:20.484788 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-cfbg4"] Nov 03 18:11:21.168100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:21.168068 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:11:21.168307 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:21.168285 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" containerID="cri-o://206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3" gracePeriod=30 Nov 03 18:11:22.297469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:22.297425 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" path="/var/lib/kubelet/pods/7e7909f6-bccc-462b-9c42-fc120e4dc653/volumes" Nov 03 18:11:26.448454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.448426 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:11:26.570364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570332 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570386 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570423 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570466 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570508 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570539 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570725 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.570909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570763 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:26.570909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570763 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:26.570909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570859 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:26.570909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570874 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.571185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570933 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\") pod \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\" (UID: \"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9\") " Nov 03 18:11:26.571185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.570998 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:26.571185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571134 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:26.571869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571681 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.571869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571708 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.571869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571723 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.571869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571738 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.571869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.571752 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.573988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.573951 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:26.574143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.574014 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:26.574791 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.574756 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "kube-api-access-vgvn5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:26.574907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.574857 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo" pod "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" (UID: "bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:11:26.673235 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.673202 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.673377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.673238 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vgvn5\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-kube-api-access-vgvn5\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.673377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.673255 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:26.673377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:26.673271 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:27.275657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.275625 4980 generic.go:356] "Generic (PLEG): container finished" podID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerID="206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3" exitCode=0 Nov 03 18:11:27.275853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.275703 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" Nov 03 18:11:27.275853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.275713 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" event={"ID":"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9","Type":"ContainerDied","Data":"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3"} Nov 03 18:11:27.275853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.275814 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz" event={"ID":"bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9","Type":"ContainerDied","Data":"0f819abe0a2a249952db94c9c97c0b2d54b4deb9e255470f9e7a35d2aec15549"} Nov 03 18:11:27.275853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.275846 4980 scope.go:122] "RemoveContainer" containerID="206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3" Nov 03 18:11:27.287924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.287728 4980 scope.go:122] "RemoveContainer" containerID="206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3" Nov 03 18:11:27.288074 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:27.288052 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3\": container with ID starting with 206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3 not found: ID does not exist" containerID="206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3" Nov 03 18:11:27.288123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.288083 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3"} err="failed to get container status \"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3\": rpc error: code = NotFound desc = could not find container \"206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3\": container with ID starting with 206b536dba25d6b39aff2b987daa104de9282d9897e5efbc9735054f2f2d4bd3 not found: ID does not exist" Nov 03 18:11:27.496562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.496526 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:11:27.545663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:27.545527 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-ad914d-openshift-default-67659fdd6f-8jprz"] Nov 03 18:11:28.294791 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:28.294759 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" path="/var/lib/kubelet/pods/bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9/volumes" Nov 03 18:11:28.425102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:28.425058 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:11:28.538458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:28.538361 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:11:33.425816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.425776 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" podUID="fa811b8a-06dd-461a-b181-f67deb1780f6" containerName="payload-processing" containerID="cri-o://46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f" gracePeriod=25 Nov 03 18:11:33.540726 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.540589 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" podUID="faf9cfee-296e-4047-be51-60d31158f224" containerName="payload-pre-processing" containerID="cri-o://b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad" gracePeriod=25 Nov 03 18:11:33.545609 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.545552 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:11:33.655088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.655047 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:11:33.729541 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.729503 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:11:33.794637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.794603 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\" (UniqueName: \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\") pod \"fa811b8a-06dd-461a-b181-f67deb1780f6\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " Nov 03 18:11:33.794847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.794817 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\") pod \"fa811b8a-06dd-461a-b181-f67deb1780f6\" (UID: \"fa811b8a-06dd-461a-b181-f67deb1780f6\") " Nov 03 18:11:33.795253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.795224 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume" pod "fa811b8a-06dd-461a-b181-f67deb1780f6" (UID: "fa811b8a-06dd-461a-b181-f67deb1780f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:33.795455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.795437 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa811b8a-06dd-461a-b181-f67deb1780f6-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:33.798080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.798041 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt" pod "fa811b8a-06dd-461a-b181-f67deb1780f6" (UID: "fa811b8a-06dd-461a-b181-f67deb1780f6"). InnerVolumeSpecName "kube-api-access-4ddtt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:33.853611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.847866 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:11:33.897185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:33.897148 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ddtt\" (UniqueName: \"kubernetes.io/projected/fa811b8a-06dd-461a-b181-f67deb1780f6-kube-api-access-4ddtt\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:34.024448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.022207 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\" (UniqueName: \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\") pod \"faf9cfee-296e-4047-be51-60d31158f224\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " Nov 03 18:11:34.024448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.022422 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\") pod \"faf9cfee-296e-4047-be51-60d31158f224\" (UID: \"faf9cfee-296e-4047-be51-60d31158f224\") " Nov 03 18:11:34.038904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.038849 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume" pod "faf9cfee-296e-4047-be51-60d31158f224" (UID: "faf9cfee-296e-4047-be51-60d31158f224"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:34.039805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.039777 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn" pod "faf9cfee-296e-4047-be51-60d31158f224" (UID: "faf9cfee-296e-4047-be51-60d31158f224"). InnerVolumeSpecName "kube-api-access-dnlgn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:34.133606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.130118 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dnlgn\" (UniqueName: \"kubernetes.io/projected/faf9cfee-296e-4047-be51-60d31158f224-kube-api-access-dnlgn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:34.133606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.130170 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/faf9cfee-296e-4047-be51-60d31158f224-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:34.377709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.377675 4980 generic.go:356] "Generic (PLEG): container finished" podID="faf9cfee-296e-4047-be51-60d31158f224" containerID="b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad" exitCode=0 Nov 03 18:11:34.377907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.377727 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" Nov 03 18:11:34.377907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.377753 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" event={"ID":"faf9cfee-296e-4047-be51-60d31158f224","Type":"ContainerDied","Data":"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad"} Nov 03 18:11:34.377907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.377806 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk" event={"ID":"faf9cfee-296e-4047-be51-60d31158f224","Type":"ContainerDied","Data":"70a04b34beab80495d57dcaecba741c708be68a76ad8120ebe4c89edb089858d"} Nov 03 18:11:34.377907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.377834 4980 scope.go:122] "RemoveContainer" containerID="b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad" Nov 03 18:11:34.379349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.379326 4980 generic.go:356] "Generic (PLEG): container finished" podID="fa811b8a-06dd-461a-b181-f67deb1780f6" containerID="46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f" exitCode=0 Nov 03 18:11:34.379463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.379410 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" event={"ID":"fa811b8a-06dd-461a-b181-f67deb1780f6","Type":"ContainerDied","Data":"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f"} Nov 03 18:11:34.379463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.379449 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" event={"ID":"fa811b8a-06dd-461a-b181-f67deb1780f6","Type":"ContainerDied","Data":"49f3e5572269213fd971206a80da433fcc4c7ad312dd6e45f4b74f2bd377bdbe"} Nov 03 18:11:34.379554 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.379472 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt" Nov 03 18:11:34.389132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.389117 4980 scope.go:122] "RemoveContainer" containerID="b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad" Nov 03 18:11:34.389408 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:34.389390 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad\": container with ID starting with b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad not found: ID does not exist" containerID="b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad" Nov 03 18:11:34.389459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.389416 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad"} err="failed to get container status \"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad\": rpc error: code = NotFound desc = could not find container \"b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad\": container with ID starting with b912f508668ff5a48a42ab8a9e01a0186713ea44154f28a397fafed3338c08ad not found: ID does not exist" Nov 03 18:11:34.389459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.389436 4980 scope.go:122] "RemoveContainer" containerID="46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f" Nov 03 18:11:34.399085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.399068 4980 scope.go:122] "RemoveContainer" containerID="46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f" Nov 03 18:11:34.399336 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:34.399314 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f\": container with ID starting with 46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f not found: ID does not exist" containerID="46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f" Nov 03 18:11:34.399404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.399344 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f"} err="failed to get container status \"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f\": rpc error: code = NotFound desc = could not find container \"46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f\": container with ID starting with 46bb9406db423beebd0a94104d7997ed9e24b2fbca1dc1552fae281424ea524f not found: ID does not exist" Nov 03 18:11:34.549384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.549352 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:11:34.554134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.554071 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-b05459-9cf897d8f-hdgpt"] Nov 03 18:11:34.666084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.666055 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:11:34.672170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:34.672146 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-b05459-7b894fc567-pf2kk"] Nov 03 18:11:36.402375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:36.402338 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa811b8a-06dd-461a-b181-f67deb1780f6" path="/var/lib/kubelet/pods/fa811b8a-06dd-461a-b181-f67deb1780f6/volumes" Nov 03 18:11:36.403229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:36.403212 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="faf9cfee-296e-4047-be51-60d31158f224" path="/var/lib/kubelet/pods/faf9cfee-296e-4047-be51-60d31158f224/volumes" Nov 03 18:11:38.546303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.546269 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" podUID="f3a90b95-733c-4c85-937e-afa464c60bc5" containerName="payload-processing" containerID="cri-o://234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da" gracePeriod=25 Nov 03 18:11:38.658003 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.657963 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" podUID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" containerName="payload-pre-processing" containerID="cri-o://e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67" gracePeriod=25 Nov 03 18:11:38.837149 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.837123 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:11:38.849208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.849179 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\") pod \"f3a90b95-733c-4c85-937e-afa464c60bc5\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " Nov 03 18:11:38.849354 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.849278 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\" (UniqueName: \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\") pod \"f3a90b95-733c-4c85-937e-afa464c60bc5\" (UID: \"f3a90b95-733c-4c85-937e-afa464c60bc5\") " Nov 03 18:11:38.849863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.849821 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume" pod "f3a90b95-733c-4c85-937e-afa464c60bc5" (UID: "f3a90b95-733c-4c85-937e-afa464c60bc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:38.850016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.849979 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3a90b95-733c-4c85-937e-afa464c60bc5-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:38.852317 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.852254 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l" pod "f3a90b95-733c-4c85-937e-afa464c60bc5" (UID: "f3a90b95-733c-4c85-937e-afa464c60bc5"). InnerVolumeSpecName "kube-api-access-4s87l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:38.933088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.933064 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:11:38.951042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.951016 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\" (UniqueName: \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\") pod \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " Nov 03 18:11:38.951305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.951283 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\") pod \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\" (UID: \"571911b9-e1e4-40e8-8cea-cdfe688d7a27\") " Nov 03 18:11:38.951675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.951654 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume" pod "571911b9-e1e4-40e8-8cea-cdfe688d7a27" (UID: "571911b9-e1e4-40e8-8cea-cdfe688d7a27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:38.951827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.951814 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4s87l\" (UniqueName: \"kubernetes.io/projected/f3a90b95-733c-4c85-937e-afa464c60bc5-kube-api-access-4s87l\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:38.951873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.951831 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/571911b9-e1e4-40e8-8cea-cdfe688d7a27-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:38.953215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:38.953194 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd" pod "571911b9-e1e4-40e8-8cea-cdfe688d7a27" (UID: "571911b9-e1e4-40e8-8cea-cdfe688d7a27"). InnerVolumeSpecName "kube-api-access-rxdpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:39.052876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.052846 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxdpd\" (UniqueName: \"kubernetes.io/projected/571911b9-e1e4-40e8-8cea-cdfe688d7a27-kube-api-access-rxdpd\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:39.433389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.433358 4980 generic.go:356] "Generic (PLEG): container finished" podID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" containerID="e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67" exitCode=0 Nov 03 18:11:39.433557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.433420 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" Nov 03 18:11:39.433557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.433435 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" event={"ID":"571911b9-e1e4-40e8-8cea-cdfe688d7a27","Type":"ContainerDied","Data":"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67"} Nov 03 18:11:39.433557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.433527 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j" event={"ID":"571911b9-e1e4-40e8-8cea-cdfe688d7a27","Type":"ContainerDied","Data":"4f4baf3b1105fb3d519699764fb7586d2213eaf2f8502808fd465c8ba79382dc"} Nov 03 18:11:39.433724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.433563 4980 scope.go:122] "RemoveContainer" containerID="e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67" Nov 03 18:11:39.435272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.435252 4980 generic.go:356] "Generic (PLEG): container finished" podID="f3a90b95-733c-4c85-937e-afa464c60bc5" containerID="234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da" exitCode=0 Nov 03 18:11:39.435392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.435374 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" event={"ID":"f3a90b95-733c-4c85-937e-afa464c60bc5","Type":"ContainerDied","Data":"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da"} Nov 03 18:11:39.435441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.435414 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" event={"ID":"f3a90b95-733c-4c85-937e-afa464c60bc5","Type":"ContainerDied","Data":"7f5f1a6e91057dff22ae78b6108d34d3841078efbab9e538d3cbc84da4894bcf"} Nov 03 18:11:39.435479 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.435441 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l" Nov 03 18:11:39.448240 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.448218 4980 scope.go:122] "RemoveContainer" containerID="e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67" Nov 03 18:11:39.448503 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:39.448484 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67\": container with ID starting with e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67 not found: ID does not exist" containerID="e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67" Nov 03 18:11:39.448616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.448513 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67"} err="failed to get container status \"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67\": rpc error: code = NotFound desc = could not find container \"e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67\": container with ID starting with e709d7f987bb3df1a1f62f9ea6e5d01995514d371fb46de77a78f5eb67320e67 not found: ID does not exist" Nov 03 18:11:39.448616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.448530 4980 scope.go:122] "RemoveContainer" containerID="234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da" Nov 03 18:11:39.458723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.458707 4980 scope.go:122] "RemoveContainer" containerID="234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da" Nov 03 18:11:39.459014 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:39.458991 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da\": container with ID starting with 234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da not found: ID does not exist" containerID="234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da" Nov 03 18:11:39.459096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.459021 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da"} err="failed to get container status \"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da\": rpc error: code = NotFound desc = could not find container \"234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da\": container with ID starting with 234eefdf22200502691bd4b356bdc4ce8810d9eb65a286f715125e4817b7f0da not found: ID does not exist" Nov 03 18:11:39.603987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.603956 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:11:39.609633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.609599 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w6-2569c8-6d497bc985-xfs2l"] Nov 03 18:11:39.720032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.719998 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:11:39.723552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.723532 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:11:39.728482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.728458 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w6-2569c8-6c68df7b5f-gkk7j"] Nov 03 18:11:39.835563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:39.835526 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:11:40.344442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:40.344341 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" path="/var/lib/kubelet/pods/571911b9-e1e4-40e8-8cea-cdfe688d7a27/volumes" Nov 03 18:11:40.345208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:40.345184 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3a90b95-733c-4c85-937e-afa464c60bc5" path="/var/lib/kubelet/pods/f3a90b95-733c-4c85-937e-afa464c60bc5/volumes" Nov 03 18:11:41.654254 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:41.654214 4980 scope.go:122] "RemoveContainer" containerID="00ea6009572e958a1885efc070f9f9688794c6d6e74ff2025c862738c836e034" Nov 03 18:11:44.520280 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:44.520244 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:11:44.720735 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:44.720691 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" podUID="bb0d75fc-5701-4616-b748-bbaba2b388ee" containerName="payload-processing" containerID="cri-o://06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134" gracePeriod=25 Nov 03 18:11:44.836166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:44.836132 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" podUID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" containerName="payload-pre-processing" containerID="cri-o://97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d" gracePeriod=25 Nov 03 18:11:45.005498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.005472 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:11:45.087691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.087617 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\") pod \"bb0d75fc-5701-4616-b748-bbaba2b388ee\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " Nov 03 18:11:45.087838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.087700 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\" (UniqueName: \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\") pod \"bb0d75fc-5701-4616-b748-bbaba2b388ee\" (UID: \"bb0d75fc-5701-4616-b748-bbaba2b388ee\") " Nov 03 18:11:45.088088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.088060 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume" pod "bb0d75fc-5701-4616-b748-bbaba2b388ee" (UID: "bb0d75fc-5701-4616-b748-bbaba2b388ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:45.088395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.088368 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb0d75fc-5701-4616-b748-bbaba2b388ee-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:45.090368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.090339 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7" pod "bb0d75fc-5701-4616-b748-bbaba2b388ee" (UID: "bb0d75fc-5701-4616-b748-bbaba2b388ee"). InnerVolumeSpecName "kube-api-access-vh2g7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:45.106832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.106812 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:11:45.189555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.189524 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\") pod \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " Nov 03 18:11:45.189798 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.189777 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\" (UniqueName: \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\") pod \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\" (UID: \"fea890b4-d192-4ac3-844c-f9b8f9cdacca\") " Nov 03 18:11:45.189934 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.189912 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume" pod "fea890b4-d192-4ac3-844c-f9b8f9cdacca" (UID: "fea890b4-d192-4ac3-844c-f9b8f9cdacca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:45.190285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.190254 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vh2g7\" (UniqueName: \"kubernetes.io/projected/bb0d75fc-5701-4616-b748-bbaba2b388ee-kube-api-access-vh2g7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:45.190285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.190284 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fea890b4-d192-4ac3-844c-f9b8f9cdacca-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:45.191954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.191933 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx" pod "fea890b4-d192-4ac3-844c-f9b8f9cdacca" (UID: "fea890b4-d192-4ac3-844c-f9b8f9cdacca"). InnerVolumeSpecName "kube-api-access-bskcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:45.291255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.291208 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskcx\" (UniqueName: \"kubernetes.io/projected/fea890b4-d192-4ac3-844c-f9b8f9cdacca-kube-api-access-bskcx\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:45.540423 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.540389 4980 generic.go:356] "Generic (PLEG): container finished" podID="bb0d75fc-5701-4616-b748-bbaba2b388ee" containerID="06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134" exitCode=0 Nov 03 18:11:45.540937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.540432 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" Nov 03 18:11:45.540937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.540475 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" event={"ID":"bb0d75fc-5701-4616-b748-bbaba2b388ee","Type":"ContainerDied","Data":"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134"} Nov 03 18:11:45.540937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.540530 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g" event={"ID":"bb0d75fc-5701-4616-b748-bbaba2b388ee","Type":"ContainerDied","Data":"f7b13b0cb4ea84dc138220efb39e1bdc22d71ec398f64e197440a3428b54a0aa"} Nov 03 18:11:45.540937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.540552 4980 scope.go:122] "RemoveContainer" containerID="06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134" Nov 03 18:11:45.541938 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.541915 4980 generic.go:356] "Generic (PLEG): container finished" podID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" containerID="97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d" exitCode=0 Nov 03 18:11:45.542064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.541938 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" event={"ID":"fea890b4-d192-4ac3-844c-f9b8f9cdacca","Type":"ContainerDied","Data":"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d"} Nov 03 18:11:45.542064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.541969 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" Nov 03 18:11:45.542064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.541981 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc" event={"ID":"fea890b4-d192-4ac3-844c-f9b8f9cdacca","Type":"ContainerDied","Data":"69ca5434c35076c0223bd46f0a6329e43a78a5af5609ed482f3c46426a75b1a2"} Nov 03 18:11:45.550716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.550696 4980 scope.go:122] "RemoveContainer" containerID="06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134" Nov 03 18:11:45.551015 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:45.550996 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134\": container with ID starting with 06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134 not found: ID does not exist" containerID="06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134" Nov 03 18:11:45.551120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.551024 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134"} err="failed to get container status \"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134\": rpc error: code = NotFound desc = could not find container \"06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134\": container with ID starting with 06aa5d2c4b944ca1d4a7eeb85854e3fa304cd9854b4020a7eed02b0c88cb7134 not found: ID does not exist" Nov 03 18:11:45.551120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.551046 4980 scope.go:122] "RemoveContainer" containerID="97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d" Nov 03 18:11:45.560468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.560448 4980 scope.go:122] "RemoveContainer" containerID="97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d" Nov 03 18:11:45.560742 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:45.560722 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d\": container with ID starting with 97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d not found: ID does not exist" containerID="97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d" Nov 03 18:11:45.560800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.560750 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d"} err="failed to get container status \"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d\": rpc error: code = NotFound desc = could not find container \"97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d\": container with ID starting with 97045b2bd410161207d4bf1f6bc606999d51606dd9efe55c50f12313babf925d not found: ID does not exist" Nov 03 18:11:45.710232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.710199 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:11:45.760141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.760108 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-19d275-67c78b557f-n7j5g"] Nov 03 18:11:45.834137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.834057 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:11:45.841466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:45.841435 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-19d275-75c4654d5f-lgrrc"] Nov 03 18:11:46.296263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:46.296234 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0d75fc-5701-4616-b748-bbaba2b388ee" path="/var/lib/kubelet/pods/bb0d75fc-5701-4616-b748-bbaba2b388ee/volumes" Nov 03 18:11:46.296652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:46.296637 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" path="/var/lib/kubelet/pods/fea890b4-d192-4ac3-844c-f9b8f9cdacca/volumes" Nov 03 18:11:47.261444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:47.261414 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:11:47.261867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:47.261661 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" containerID="cri-o://43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3" gracePeriod=30 Nov 03 18:11:49.969106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:49.969071 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:11:52.614427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.614391 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:11:52.643794 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.643768 4980 generic.go:356] "Generic (PLEG): container finished" podID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerID="43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3" exitCode=0 Nov 03 18:11:52.643937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.643817 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" Nov 03 18:11:52.643937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.643856 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerDied","Data":"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3"} Nov 03 18:11:52.643937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.643907 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44" event={"ID":"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f","Type":"ContainerDied","Data":"01d0651e770939ed79dc6be17f2647d891465ca4b31839105bf78d3fa00a07a1"} Nov 03 18:11:52.643937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.643931 4980 scope.go:122] "RemoveContainer" containerID="43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3" Nov 03 18:11:52.654353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.654329 4980 scope.go:122] "RemoveContainer" containerID="480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169" Nov 03 18:11:52.691010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.690990 4980 scope.go:122] "RemoveContainer" containerID="43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3" Nov 03 18:11:52.691352 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:52.691332 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3\": container with ID starting with 43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3 not found: ID does not exist" containerID="43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3" Nov 03 18:11:52.691444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.691368 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3"} err="failed to get container status \"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3\": rpc error: code = NotFound desc = could not find container \"43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3\": container with ID starting with 43f9d27685cf97b3c73750506a2e5e76911a4938ce281dae69e85f8052ffa2e3 not found: ID does not exist" Nov 03 18:11:52.691444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.691396 4980 scope.go:122] "RemoveContainer" containerID="480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169" Nov 03 18:11:52.691695 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:52.691672 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169\": container with ID starting with 480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169 not found: ID does not exist" containerID="480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169" Nov 03 18:11:52.691751 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.691701 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169"} err="failed to get container status \"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169\": rpc error: code = NotFound desc = could not find container \"480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169\": container with ID starting with 480d6df0408e67d52e14aa902de5c560f4a1b1f9cdcc4578e6be2df666b0e169 not found: ID does not exist" Nov 03 18:11:52.789783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.789750 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.789985 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.789818 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.789985 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.789894 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.789985 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.789962 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790013 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790105 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790146 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:52.790257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790199 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:52.790257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790203 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790287 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790357 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\") pod \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\" (UID: \"6c049c75-fe72-4c2c-b675-cb5a1ee3c70f\") " Nov 03 18:11:52.790437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790371 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:52.790622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790562 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:52.790765 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.790742 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:52.791644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.791336 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.791644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.791366 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.791644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.791380 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.791644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.791393 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.791644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.791405 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.792613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.792565 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:11:52.792836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.792814 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:11:52.792974 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.792955 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:52.793028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.792983 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4" pod "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" (UID: "6c049c75-fe72-4c2c-b675-cb5a1ee3c70f"). InnerVolumeSpecName "kube-api-access-wtvk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:52.893294 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.893193 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtvk4\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-kube-api-access-wtvk4\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.893294 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.893233 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.893294 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.893249 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:52.893294 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:52.893261 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:53.083986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:53.083936 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:11:53.088055 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:53.088027 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-e75517-openshift-default-b89796c75-r7c44"] Nov 03 18:11:54.452280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:54.452242 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" path="/var/lib/kubelet/pods/6c049c75-fe72-4c2c-b675-cb5a1ee3c70f/volumes" Nov 03 18:11:54.979601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:54.978794 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" podUID="9141a3f7-7938-4d70-a062-edec28a48a1c" containerName="payload-processing" containerID="cri-o://5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf" gracePeriod=25 Nov 03 18:11:55.316034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.316010 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:11:55.434710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.434679 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\" (UniqueName: \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\") pod \"9141a3f7-7938-4d70-a062-edec28a48a1c\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " Nov 03 18:11:55.434915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.434840 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\") pod \"9141a3f7-7938-4d70-a062-edec28a48a1c\" (UID: \"9141a3f7-7938-4d70-a062-edec28a48a1c\") " Nov 03 18:11:55.435232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.435206 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume" pod "9141a3f7-7938-4d70-a062-edec28a48a1c" (UID: "9141a3f7-7938-4d70-a062-edec28a48a1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:11:55.437051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.437023 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq" pod "9141a3f7-7938-4d70-a062-edec28a48a1c" (UID: "9141a3f7-7938-4d70-a062-edec28a48a1c"). InnerVolumeSpecName "kube-api-access-9hkhq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:11:55.537013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.536923 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hkhq\" (UniqueName: \"kubernetes.io/projected/9141a3f7-7938-4d70-a062-edec28a48a1c-kube-api-access-9hkhq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:55.537013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.536960 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9141a3f7-7938-4d70-a062-edec28a48a1c-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:11:55.692525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.692490 4980 generic.go:356] "Generic (PLEG): container finished" podID="9141a3f7-7938-4d70-a062-edec28a48a1c" containerID="5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf" exitCode=0 Nov 03 18:11:55.692836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.692587 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" Nov 03 18:11:55.692931 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.692596 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" event={"ID":"9141a3f7-7938-4d70-a062-edec28a48a1c","Type":"ContainerDied","Data":"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf"} Nov 03 18:11:55.692994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.692973 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2" event={"ID":"9141a3f7-7938-4d70-a062-edec28a48a1c","Type":"ContainerDied","Data":"2412080883de35d658b539c4fce248b7402fa721a340792e51cfc9e92aa2f4cf"} Nov 03 18:11:55.693227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.693210 4980 scope.go:122] "RemoveContainer" containerID="5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf" Nov 03 18:11:55.718335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.718303 4980 scope.go:122] "RemoveContainer" containerID="5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf" Nov 03 18:11:55.718664 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:11:55.718644 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf\": container with ID starting with 5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf not found: ID does not exist" containerID="5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf" Nov 03 18:11:55.718787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.718768 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf"} err="failed to get container status \"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf\": rpc error: code = NotFound desc = could not find container \"5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf\": container with ID starting with 5790f8f09e5d10c66809a4189373b98ccac0cb3f789ca73cc8070cd397ffa0cf not found: ID does not exist" Nov 03 18:11:55.885654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.882707 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:11:55.913943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:55.913906 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f5100c-666cbb9fb9-5fvd2"] Nov 03 18:11:56.310602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:11:56.307856 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9141a3f7-7938-4d70-a062-edec28a48a1c" path="/var/lib/kubelet/pods/9141a3f7-7938-4d70-a062-edec28a48a1c/volumes" Nov 03 18:12:01.211424 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:01.209098 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 18:12:01.221311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:01.221268 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" containerID="cri-o://b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504" gracePeriod=30 Nov 03 18:12:02.745929 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.745895 4980 generic.go:356] "Generic (PLEG): container finished" podID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerID="38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29" exitCode=0 Nov 03 18:12:02.746376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.745970 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerDied","Data":"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29"} Nov 03 18:12:02.746376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.746367 4980 scope.go:122] "RemoveContainer" containerID="38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29" Nov 03 18:12:02.748047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.748010 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:02.748163 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.748084 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:02.748163 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:02.748119 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:03.779731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:03.767502 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerStarted","Data":"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61"} Nov 03 18:12:04.105870 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.105788 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:12:04.107283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.107250 4980 patch_prober.go:29] interesting pod/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 18:12:04.107435 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.107305 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 18:12:04.397422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.397345 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:12:04.677656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.672685 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:12:04.787141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:04.787105 4980 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" secret="" err="secret \"e2e-api-a-f5100c-openshift-default-dockercfg-hpszq\" not found" Nov 03 18:12:05.105851 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:05.105821 4980 patch_prober.go:29] interesting pod/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" start-of-body= Nov 03 18:12:05.106044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:05.105874 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.142:15021/healthz/ready\": dial tcp 10.132.0.142:15021: connect: connection refused" Nov 03 18:12:05.793209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:05.793170 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" containerID="cri-o://5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61" gracePeriod=30 Nov 03 18:12:06.694843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.694819 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 18:12:06.803452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.803420 4980 generic.go:356] "Generic (PLEG): container finished" podID="17572545-3b6d-4d65-83a9-968c02a960c9" containerID="b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504" exitCode=0 Nov 03 18:12:06.803980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.803474 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" Nov 03 18:12:06.803980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.803510 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" event={"ID":"17572545-3b6d-4d65-83a9-968c02a960c9","Type":"ContainerDied","Data":"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504"} Nov 03 18:12:06.803980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.803611 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf" event={"ID":"17572545-3b6d-4d65-83a9-968c02a960c9","Type":"ContainerDied","Data":"033eec56a6615d781bbab7fb0056db1d45b62fa84cea57bfb60acca5dfaf88c4"} Nov 03 18:12:06.803980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.803648 4980 scope.go:122] "RemoveContainer" containerID="b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504" Nov 03 18:12:06.808555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.808495 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.808743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.808638 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.809129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.809074 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:06.809659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.809633 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:06.809766 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.809680 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.809825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.809776 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.810509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.809806 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.810726 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.810707 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.811153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.811039 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:06.811369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.811350 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.811803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.811689 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:06.811945 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.811928 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.812121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.812103 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\") pod \"17572545-3b6d-4d65-83a9-968c02a960c9\" (UID: \"17572545-3b6d-4d65-83a9-968c02a960c9\") " Nov 03 18:12:06.812397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.812371 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:06.813034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.813011 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.813169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.813154 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17572545-3b6d-4d65-83a9-968c02a960c9-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.813352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.813333 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.813461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.813448 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.813556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.813544 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.814432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.814403 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:06.814936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.814759 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "kube-api-access-x6fkl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:06.814936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.814902 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:06.818681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.818449 4980 scope.go:122] "RemoveContainer" containerID="b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504" Nov 03 18:12:06.818916 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:06.818886 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504\": container with ID starting with b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504 not found: ID does not exist" containerID="b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504" Nov 03 18:12:06.819012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.818929 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504"} err="failed to get container status \"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504\": rpc error: code = NotFound desc = could not find container \"b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504\": container with ID starting with b27e75d59bc2c8c6b8f42187ee125c645ab5e3a7593100d9e0e4bfd9af02e504 not found: ID does not exist" Nov 03 18:12:06.819822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.819797 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy" pod "17572545-3b6d-4d65-83a9-968c02a960c9" (UID: "17572545-3b6d-4d65-83a9-968c02a960c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:06.914995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.914912 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6fkl\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-kube-api-access-x6fkl\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.914995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.914943 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17572545-3b6d-4d65-83a9-968c02a960c9-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.914995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.914954 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17572545-3b6d-4d65-83a9-968c02a960c9-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:06.914995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:06.914962 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17572545-3b6d-4d65-83a9-968c02a960c9-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:07.293599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.288621 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 18:12:07.310597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.305499 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-ldqtf"] Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.387656 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388638 4980 cpu_manager.go:414] "containerMap: removing container" podUID="bb0d75fc-5701-4616-b748-bbaba2b388ee" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388663 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0d75fc-5701-4616-b748-bbaba2b388ee" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388691 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9141a3f7-7938-4d70-a062-edec28a48a1c" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388700 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9141a3f7-7938-4d70-a062-edec28a48a1c" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388714 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6432b682-dbb3-493b-b52f-374822e176c7" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388722 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6432b682-dbb3-493b-b52f-374822e176c7" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388739 4980 cpu_manager.go:414] "containerMap: removing container" podUID="67767ae8-7eab-4610-968f-2511f3569102" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388745 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67767ae8-7eab-4610-968f-2511f3569102" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388757 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388764 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388781 4980 cpu_manager.go:414] "containerMap: removing container" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388788 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388804 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f3a90b95-733c-4c85-937e-afa464c60bc5" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388811 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3a90b95-733c-4c85-937e-afa464c60bc5" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388828 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388841 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388853 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388860 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388873 4980 cpu_manager.go:414] "containerMap: removing container" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388882 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388897 4980 cpu_manager.go:414] "containerMap: removing container" podUID="eb441d60-e513-41d0-b889-17964b4d2ac3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388904 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb441d60-e513-41d0-b889-17964b4d2ac3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388913 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f047702e-2c8d-4ca3-9526-7067055dc854" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388920 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f047702e-2c8d-4ca3-9526-7067055dc854" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388935 4980 cpu_manager.go:414] "containerMap: removing container" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388942 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388955 4980 cpu_manager.go:414] "containerMap: removing container" podUID="07107523-9d65-4545-a3f6-26474bf83639" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388964 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07107523-9d65-4545-a3f6-26474bf83639" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388979 4980 cpu_manager.go:414] "containerMap: removing container" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.388986 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389004 4980 cpu_manager.go:414] "containerMap: removing container" podUID="faf9cfee-296e-4047-be51-60d31158f224" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389011 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="faf9cfee-296e-4047-be51-60d31158f224" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389054 4980 cpu_manager.go:414] "containerMap: removing container" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389063 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389088 4980 cpu_manager.go:414] "containerMap: removing container" podUID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389096 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389108 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389116 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389132 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3e8378a5-4ac6-4726-81ae-070a3937d984" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389139 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e8378a5-4ac6-4726-81ae-070a3937d984" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389153 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fa811b8a-06dd-461a-b181-f67deb1780f6" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389160 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa811b8a-06dd-461a-b181-f67deb1780f6" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389171 4980 cpu_manager.go:414] "containerMap: removing container" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389179 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389352 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0d75fc-5701-4616-b748-bbaba2b388ee" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389366 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9141a3f7-7938-4d70-a062-edec28a48a1c" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389381 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6432b682-dbb3-493b-b52f-374822e176c7" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389395 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="67767ae8-7eab-4610-968f-2511f3569102" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389408 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389418 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7909f6-bccc-462b-9c42-fc120e4dc653" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389429 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3a90b95-733c-4c85-937e-afa464c60bc5" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389440 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fea890b4-d192-4ac3-844c-f9b8f9cdacca" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389450 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c049c75-fe72-4c2c-b675-cb5a1ee3c70f" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389459 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389468 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb441d60-e513-41d0-b889-17964b4d2ac3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389476 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f047702e-2c8d-4ca3-9526-7067055dc854" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389486 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="07531d91-a805-4751-aa5e-5aca33cbccdc" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389494 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="07107523-9d65-4545-a3f6-26474bf83639" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389503 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea5267ea-f827-4810-a5bb-93032a869f8e" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389514 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="faf9cfee-296e-4047-be51-60d31158f224" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389524 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" containerName="istio-proxy" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389535 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="571911b9-e1e4-40e8-8cea-cdfe688d7a27" containerName="payload-pre-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389544 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a0fadc6-b470-4f2a-ae09-889668bfcdc3" containerName="maas-api" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389552 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e8378a5-4ac6-4726-81ae-070a3937d984" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389560 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa811b8a-06dd-461a-b181-f67deb1780f6" containerName="payload-processing" Nov 03 18:12:07.395599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.389592 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd2f3bdf-65d7-4bf2-9d75-d1e6dd066ec9" containerName="istio-proxy" Nov 03 18:12:07.419669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.417869 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458220 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458301 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458482 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458524 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458605 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458650 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458686 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5htrp\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458788 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.465847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.458846 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.515717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.515312 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.572857 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-ca6b33-openshift-default-dockercfg-vl8nw\"" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576473 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576531 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576625 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576658 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576703 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576743 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576774 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5htrp\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576797 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.587288 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.576838 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.614680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.595161 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.614680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.595437 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.614680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.596049 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.614680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.614411 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.621099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.621053 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.737014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.736948 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.737462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.737406 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.740809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.740777 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5htrp\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.741143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.741120 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\") pod \"e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:07.773279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:07.773002 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:08.282866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.282831 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:12:08.290319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.290278 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:08.290452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.290381 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:08.290452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.290433 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:08.296335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.296313 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17572545-3b6d-4d65-83a9-968c02a960c9" path="/var/lib/kubelet/pods/17572545-3b6d-4d65-83a9-968c02a960c9/volumes" Nov 03 18:12:08.844331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.844294 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerStarted","Data":"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888"} Nov 03 18:12:08.844566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.844352 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerStarted","Data":"17d65d95d098fb1569bb2c3570f8aca4e53e6a354f497664f26b265a880c1d1a"} Nov 03 18:12:08.979680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:08.970304 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podStartSLOduration=1.970285609 podStartE2EDuration="1.970285609s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:08.968060381 +0000 UTC m=+2308.797143983" watchObservedRunningTime="2026-11-03 18:12:08.970285609 +0000 UTC m=+2308.799369212" Nov 03 18:12:09.774503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:09.774470 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:09.780410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:09.780382 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:09.871712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:09.853392 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:09.871712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:09.854664 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:12:14.470443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.470413 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:12:14.521854 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:14.521820 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:12:14.592364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592284 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592346 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592437 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592497 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592524 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592637 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:14.592724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592663 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592719 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592753 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592816 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\") pod \"727ec5a8-b3d9-460d-afcc-342f989f21e3\" (UID: \"727ec5a8-b3d9-460d-afcc-342f989f21e3\") " Nov 03 18:12:14.592892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592833 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:14.593058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.592940 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:14.593190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593163 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:14.593371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593349 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:14.593723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593702 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.593800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593731 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.593800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593746 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.593800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593761 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.593800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.593775 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/727ec5a8-b3d9-460d-afcc-342f989f21e3-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.595118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.595093 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:14.595191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.595161 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:14.595340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.595315 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:14.595412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.595397 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj" pod "727ec5a8-b3d9-460d-afcc-342f989f21e3" (UID: "727ec5a8-b3d9-460d-afcc-342f989f21e3"). InnerVolumeSpecName "kube-api-access-cszwj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:14.695463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.695428 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszwj\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-kube-api-access-cszwj\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.695463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.695461 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.695691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.695471 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.695691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.695480 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/727ec5a8-b3d9-460d-afcc-342f989f21e3-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:14.890078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.889992 4980 generic.go:356] "Generic (PLEG): container finished" podID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerID="5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61" exitCode=0 Nov 03 18:12:14.890078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.890043 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" Nov 03 18:12:14.890311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.890075 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerDied","Data":"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61"} Nov 03 18:12:14.890311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.890140 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt" event={"ID":"727ec5a8-b3d9-460d-afcc-342f989f21e3","Type":"ContainerDied","Data":"b43a3e2508776401e6d8d3be1800cb852d8bfeff885e7c718144253ed04f8adf"} Nov 03 18:12:14.890311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.890168 4980 scope.go:122] "RemoveContainer" containerID="5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61" Nov 03 18:12:14.901999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.901978 4980 scope.go:122] "RemoveContainer" containerID="38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29" Nov 03 18:12:14.912959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.912659 4980 scope.go:122] "RemoveContainer" containerID="5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61" Nov 03 18:12:14.913298 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:14.913273 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61\": container with ID starting with 5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61 not found: ID does not exist" containerID="5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61" Nov 03 18:12:14.913411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.913308 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61"} err="failed to get container status \"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61\": rpc error: code = NotFound desc = could not find container \"5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61\": container with ID starting with 5bc45bea54afe1a7b634c1f8192b5e3ebd32519de9dab71fc0a170ab773ddb61 not found: ID does not exist" Nov 03 18:12:14.913411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.913328 4980 scope.go:122] "RemoveContainer" containerID="38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29" Nov 03 18:12:14.913686 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:14.913666 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29\": container with ID starting with 38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29 not found: ID does not exist" containerID="38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29" Nov 03 18:12:14.913742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:14.913693 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29"} err="failed to get container status \"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29\": rpc error: code = NotFound desc = could not find container \"38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29\": container with ID starting with 38ca4dae43f9bcbc29e365b801d9a3fe9293d0a37087aaccf0cb2173ab621c29 not found: ID does not exist" Nov 03 18:12:15.063770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:15.063732 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:12:15.071898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:15.071862 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f5100c-openshift-default-8698ccc79f-htgrt"] Nov 03 18:12:16.404591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.402541 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" path="/var/lib/kubelet/pods/727ec5a8-b3d9-460d-afcc-342f989f21e3/volumes" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.434229 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435243 4980 cpu_manager.go:414] "containerMap: removing container" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435264 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435296 4980 cpu_manager.go:414] "containerMap: removing container" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435305 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435470 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.435740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.435491 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="727ec5a8-b3d9-460d-afcc-342f989f21e3" containerName="istio-proxy" Nov 03 18:12:16.450090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.446867 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.462718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.462688 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:12:16.508673 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.506893 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-ca6b33-dockercfg-92bln\"" Nov 03 18:12:16.508673 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.507624 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-ca6b33\"" Nov 03 18:12:16.616892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.602084 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vktkv\" (UniqueName: \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.616892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.613287 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.724669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.721117 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:12:16.733873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.733845 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:12:16.734171 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.734151 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:16.776352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.776318 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.776542 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.776508 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vktkv\" (UniqueName: \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.777139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.777107 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.839848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.839816 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vktkv\" (UniqueName: \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\") pod \"payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:16.878320 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.878281 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:16.878525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.878472 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cj29\" (UniqueName: \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:16.980275 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.980183 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:16.980275 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.980260 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cj29\" (UniqueName: \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:16.981118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:16.981087 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:17.043556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.043529 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cj29\" (UniqueName: \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\") pod \"payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:17.055593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.055443 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:17.083747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.083719 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:17.452447 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.452415 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:12:17.453379 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:12:17.453346 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ecb4643_e2bf_4f7e_a110_f0927207496f.slice/crio-fd7a090d8f8ce1d0802766c9f54c529a32e534f3354c786f63732caf87688c0f WatchSource:0}: Error finding container fd7a090d8f8ce1d0802766c9f54c529a32e534f3354c786f63732caf87688c0f: Status 404 returned error can't find the container with id fd7a090d8f8ce1d0802766c9f54c529a32e534f3354c786f63732caf87688c0f Nov 03 18:12:17.468563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.468536 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:12:17.468797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.468772 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" podUID="633a5340-a940-4657-b45f-16a2dddf9233" containerName="maas-api" containerID="cri-o://c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908" gracePeriod=30 Nov 03 18:12:17.513347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.513313 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:12:17.554081 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:12:17.554047 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod090d5579_9eed_4f9d_94d5_821b10da2536.slice/crio-a288a50eb42d2fec2d00aee09b285352f940bd4ddd60904799a528fd9cd55b2c WatchSource:0}: Error finding container a288a50eb42d2fec2d00aee09b285352f940bd4ddd60904799a528fd9cd55b2c: Status 404 returned error can't find the container with id a288a50eb42d2fec2d00aee09b285352f940bd4ddd60904799a528fd9cd55b2c Nov 03 18:12:17.800435 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.800409 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:12:17.839493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.839462 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\") pod \"633a5340-a940-4657-b45f-16a2dddf9233\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " Nov 03 18:12:17.839684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.839605 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\") pod \"633a5340-a940-4657-b45f-16a2dddf9233\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " Nov 03 18:12:17.839786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.839757 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\" (UniqueName: \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\") pod \"633a5340-a940-4657-b45f-16a2dddf9233\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " Nov 03 18:12:17.839851 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.839819 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\") pod \"633a5340-a940-4657-b45f-16a2dddf9233\" (UID: \"633a5340-a940-4657-b45f-16a2dddf9233\") " Nov 03 18:12:17.841502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.841446 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca" pod "633a5340-a940-4657-b45f-16a2dddf9233" (UID: "633a5340-a940-4657-b45f-16a2dddf9233"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:17.842236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.842204 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs" pod "633a5340-a940-4657-b45f-16a2dddf9233" (UID: "633a5340-a940-4657-b45f-16a2dddf9233"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:12:17.842361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.842298 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb" pod "633a5340-a940-4657-b45f-16a2dddf9233" (UID: "633a5340-a940-4657-b45f-16a2dddf9233"). InnerVolumeSpecName "kube-api-access-npxhb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:17.843422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.843400 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls" pod "633a5340-a940-4657-b45f-16a2dddf9233" (UID: "633a5340-a940-4657-b45f-16a2dddf9233"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:12:17.914586 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.914534 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" event={"ID":"6ecb4643-e2bf-4f7e-a110-f0927207496f","Type":"ContainerStarted","Data":"300aeb2feace0c23f56ccd3897a705353233fd67d79e66f1316b50b101b011fb"} Nov 03 18:12:17.914733 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.914611 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:17.914733 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.914632 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" event={"ID":"6ecb4643-e2bf-4f7e-a110-f0927207496f","Type":"ContainerStarted","Data":"fd7a090d8f8ce1d0802766c9f54c529a32e534f3354c786f63732caf87688c0f"} Nov 03 18:12:17.915942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.915913 4980 generic.go:356] "Generic (PLEG): container finished" podID="633a5340-a940-4657-b45f-16a2dddf9233" containerID="c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908" exitCode=0 Nov 03 18:12:17.916079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.915953 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" Nov 03 18:12:17.916079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.915999 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" event={"ID":"633a5340-a940-4657-b45f-16a2dddf9233","Type":"ContainerDied","Data":"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908"} Nov 03 18:12:17.916079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.916040 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2" event={"ID":"633a5340-a940-4657-b45f-16a2dddf9233","Type":"ContainerDied","Data":"ccb15e07134b8c9ebda64eaff02beb1305b544fb39bc5110936c76b80c13838f"} Nov 03 18:12:17.916079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.916062 4980 scope.go:122] "RemoveContainer" containerID="c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908" Nov 03 18:12:17.917665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.917642 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" event={"ID":"090d5579-9eed-4f9d-94d5-821b10da2536","Type":"ContainerStarted","Data":"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7"} Nov 03 18:12:17.917764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.917687 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" event={"ID":"090d5579-9eed-4f9d-94d5-821b10da2536","Type":"ContainerStarted","Data":"a288a50eb42d2fec2d00aee09b285352f940bd4ddd60904799a528fd9cd55b2c"} Nov 03 18:12:17.917764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.917711 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.945002 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.945084 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/633a5340-a940-4657-b45f-16a2dddf9233-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.945104 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npxhb\" (UniqueName: \"kubernetes.io/projected/633a5340-a940-4657-b45f-16a2dddf9233-kube-api-access-npxhb\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.945122 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/633a5340-a940-4657-b45f-16a2dddf9233-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.948217 4980 scope.go:122] "RemoveContainer" containerID="c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:17.948678 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908\": container with ID starting with c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908 not found: ID does not exist" containerID="c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908" Nov 03 18:12:17.949621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:17.948714 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908"} err="failed to get container status \"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908\": rpc error: code = NotFound desc = could not find container \"c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908\": container with ID starting with c543842a8521a9f6d93ea4ff8ba58c673201972798ff731d26faac26501c0908 not found: ID does not exist" Nov 03 18:12:18.060397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:18.060288 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" podStartSLOduration=2.06026786 podStartE2EDuration="2.06026786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:18.059206486 +0000 UTC m=+2317.888290089" watchObservedRunningTime="2026-11-03 18:12:18.06026786 +0000 UTC m=+2317.889351463" Nov 03 18:12:18.184818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:18.182644 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" podStartSLOduration=2.18262558 podStartE2EDuration="2.18262558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:18.181965321 +0000 UTC m=+2318.011048924" watchObservedRunningTime="2026-11-03 18:12:18.18262558 +0000 UTC m=+2318.011709184" Nov 03 18:12:18.369356 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:18.369280 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:12:18.373975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:18.373951 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-8dfc64-679bd4dc4f-mc4z2"] Nov 03 18:12:20.294899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:20.294864 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="633a5340-a940-4657-b45f-16a2dddf9233" path="/var/lib/kubelet/pods/633a5340-a940-4657-b45f-16a2dddf9233/volumes" Nov 03 18:12:28.890613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:28.890552 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:12:28.891481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:28.891445 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" containerID="cri-o://31b9ae168fcd3848b7bb63a3e3a27aaf5afc3efbeb2435e0e63bd8820b43fe11" gracePeriod=30 Nov 03 18:12:28.928615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:28.928554 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:12:28.928775 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:28.928660 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:12:34.141199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.141165 4980 generic.go:356] "Generic (PLEG): container finished" podID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerID="31b9ae168fcd3848b7bb63a3e3a27aaf5afc3efbeb2435e0e63bd8820b43fe11" exitCode=0 Nov 03 18:12:34.141920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.141230 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerDied","Data":"31b9ae168fcd3848b7bb63a3e3a27aaf5afc3efbeb2435e0e63bd8820b43fe11"} Nov 03 18:12:34.141920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.141305 4980 scope.go:122] "RemoveContainer" containerID="7da59dedf7ce897534bc5d6fce1d53f480b52cf0c4299ad6b848d5a6282af055" Nov 03 18:12:34.185291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.185266 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:12:34.272536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272490 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272601 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272731 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272681 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272776 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272808 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272872 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.272925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272910 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.273042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.272978 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.273042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273027 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\") pod \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\" (UID: \"0f8a1b88-a32e-400e-9d3f-026254e95f4a\") " Nov 03 18:12:34.273153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273030 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:34.273153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273116 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:34.273279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273257 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:34.273802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273700 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:34.273940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273792 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:34.273940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273838 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.274025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273945 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.274025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.273965 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.275304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.275280 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:34.275304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.275298 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:34.275784 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.275770 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:34.275850 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.275818 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2" pod "0f8a1b88-a32e-400e-9d3f-026254e95f4a" (UID: "0f8a1b88-a32e-400e-9d3f-026254e95f4a"). InnerVolumeSpecName "kube-api-access-wcvp2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:34.375120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375082 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcvp2\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-kube-api-access-wcvp2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.375120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375119 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.375336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375133 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.375336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375147 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.375336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375159 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f8a1b88-a32e-400e-9d3f-026254e95f4a-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:34.375336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:34.375170 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f8a1b88-a32e-400e-9d3f-026254e95f4a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:35.149172 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:35.149145 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" Nov 03 18:12:35.149665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:35.149168 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2" event={"ID":"0f8a1b88-a32e-400e-9d3f-026254e95f4a","Type":"ContainerDied","Data":"583b61df9c4bb0145034140c7a0a665d15aefe123b09f519bcbed7e33c03c832"} Nov 03 18:12:35.149665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:35.149228 4980 scope.go:122] "RemoveContainer" containerID="31b9ae168fcd3848b7bb63a3e3a27aaf5afc3efbeb2435e0e63bd8820b43fe11" Nov 03 18:12:35.368454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:35.368423 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:12:35.418376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:35.418294 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-8dfc64-openshift-default-74d9f44d57-ms2q2"] Nov 03 18:12:36.297383 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:36.297351 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" path="/var/lib/kubelet/pods/0f8a1b88-a32e-400e-9d3f-026254e95f4a/volumes" Nov 03 18:12:41.329392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:41.329357 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:12:44.521248 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:44.521210 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:12:46.330482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.330447 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" podUID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" containerName="payload-processing" containerID="cri-o://bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a" gracePeriod=25 Nov 03 18:12:46.618816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.618782 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:12:46.712639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.712596 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\") pod \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " Nov 03 18:12:46.712808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.712757 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\" (UniqueName: \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\") pod \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\" (UID: \"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1\") " Nov 03 18:12:46.712950 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.712927 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume" pod "40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" (UID: "40a0e75e-43b3-4166-9a6f-c3f14c8be8e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:46.713192 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.713170 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:46.715020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.714998 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d" pod "40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" (UID: "40a0e75e-43b3-4166-9a6f-c3f14c8be8e1"). InnerVolumeSpecName "kube-api-access-z679d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:46.814376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:46.814330 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z679d\" (UniqueName: \"kubernetes.io/projected/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1-kube-api-access-z679d\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:47.259942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.259901 4980 generic.go:356] "Generic (PLEG): container finished" podID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" containerID="bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a" exitCode=0 Nov 03 18:12:47.260121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.259981 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" Nov 03 18:12:47.260121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.259985 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" event={"ID":"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1","Type":"ContainerDied","Data":"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a"} Nov 03 18:12:47.260121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.260080 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb" event={"ID":"40a0e75e-43b3-4166-9a6f-c3f14c8be8e1","Type":"ContainerDied","Data":"9ee53078ceaf2019b53baaf7f177889fb154fc4b06e70ac91074d9359bc3917b"} Nov 03 18:12:47.260121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.260110 4980 scope.go:122] "RemoveContainer" containerID="bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a" Nov 03 18:12:47.271053 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.271032 4980 scope.go:122] "RemoveContainer" containerID="bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a" Nov 03 18:12:47.271327 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:47.271305 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a\": container with ID starting with bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a not found: ID does not exist" containerID="bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a" Nov 03 18:12:47.271397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.271337 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a"} err="failed to get container status \"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a\": rpc error: code = NotFound desc = could not find container \"bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a\": container with ID starting with bcf423ea04201beb02d22da980d2228b7608989f4743d60ef295cc460ff8936a not found: ID does not exist" Nov 03 18:12:47.430091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.430047 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:12:47.479684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:47.479653 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-528e53-769595b78c-4rdnb"] Nov 03 18:12:48.294162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:48.294128 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" path="/var/lib/kubelet/pods/40a0e75e-43b3-4166-9a6f-c3f14c8be8e1/volumes" Nov 03 18:12:49.196922 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.196886 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:12:49.197564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197548 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.197564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197566 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.197564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197598 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.197564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197603 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197627 4980 cpu_manager.go:414] "containerMap: removing container" podUID="633a5340-a940-4657-b45f-16a2dddf9233" containerName="maas-api" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197633 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="633a5340-a940-4657-b45f-16a2dddf9233" containerName="maas-api" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197641 4980 cpu_manager.go:414] "containerMap: removing container" podUID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" containerName="payload-processing" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197646 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" containerName="payload-processing" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197742 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="633a5340-a940-4657-b45f-16a2dddf9233" containerName="maas-api" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197751 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a0e75e-43b3-4166-9a6f-c3f14c8be8e1" containerName="payload-processing" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197761 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.197806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.197769 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f8a1b88-a32e-400e-9d3f-026254e95f4a" containerName="istio-proxy" Nov 03 18:12:49.202298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.202263 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.256993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.256954 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257041 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257077 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257185 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257224 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257350 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257382 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257633 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4ld4\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.257822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.257803 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359327 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359377 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359446 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359469 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359556 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4ld4\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359707 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359884 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359791 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359884 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359831 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.359884 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359875 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.360049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.359985 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.360398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.360373 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.360538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.360482 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.360538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.360530 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.360693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.360611 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.362448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.362422 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.362592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.362469 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.422935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.422902 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.423168 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.423145 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4ld4\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\") pod \"e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.430325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.429859 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:12:49.516551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.516453 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:49.960501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.960472 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:12:49.962352 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:12:49.962317 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29da114_deaf_4a01_ae80_90cafa3b7c24.slice/crio-9bd827750f626dad259d5c3dc74be4b683537f2d320d71b4d90837a92cba40c7 WatchSource:0}: Error finding container 9bd827750f626dad259d5c3dc74be4b683537f2d320d71b4d90837a92cba40c7: Status 404 returned error can't find the container with id 9bd827750f626dad259d5c3dc74be4b683537f2d320d71b4d90837a92cba40c7 Nov 03 18:12:49.965060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.965027 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:49.965142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.965093 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:49.965142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:49.965128 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:50.295745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:50.295662 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" event={"ID":"a29da114-deaf-4a01-ae80-90cafa3b7c24","Type":"ContainerStarted","Data":"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d"} Nov 03 18:12:50.295745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:50.295712 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" event={"ID":"a29da114-deaf-4a01-ae80-90cafa3b7c24","Type":"ContainerStarted","Data":"9bd827750f626dad259d5c3dc74be4b683537f2d320d71b4d90837a92cba40c7"} Nov 03 18:12:50.416886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:50.416832 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podStartSLOduration=1.416812979 podStartE2EDuration="1.416812979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:50.414804135 +0000 UTC m=+2350.243887737" watchObservedRunningTime="2026-11-03 18:12:50.416812979 +0000 UTC m=+2350.245896581" Nov 03 18:12:50.519596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:50.518722 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:51.315054 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:51.315019 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:12:51.315524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:51.315222 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" containerID="cri-o://a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a" gracePeriod=30 Nov 03 18:12:51.531852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:51.531812 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:51.532159 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:51.532123 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:52.517408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.517372 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:52.517899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.517429 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:52.734866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.734707 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:12:52.735068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.734993 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" containerID="cri-o://e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d" gracePeriod=30 Nov 03 18:12:52.803459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.803380 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:12:52.808066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:52.808041 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.047767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.047731 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-0bd8cf-openshift-default-dockercfg-8wnrg\"" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.049860 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.193962 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194021 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194085 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194257 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194357 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194428 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.194666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194488 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.195206 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.194678 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lprm\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.202714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.202678 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:12:53.295942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.295907 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.295955 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.295995 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296040 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296098 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296122 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296169 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296236 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lprm\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296306 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296338 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296428 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.296846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.296723 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.297028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.297005 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.297150 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.297133 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.299176 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.299152 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.299747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.299725 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.382770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.382240 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lprm\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.384056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.384008 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\") pod \"e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.423878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.423836 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:53.521154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.521079 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:53.521822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.521785 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:53.928075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.927989 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:12:53.981080 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:12:53.981039 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e4954c_e9c7_406b_80b7_83b7d0e06d93.slice/crio-ed4a6834cffe87e109563d2fc29ac32dc9d5e79a172118d741ee75476a6eb03f WatchSource:0}: Error finding container ed4a6834cffe87e109563d2fc29ac32dc9d5e79a172118d741ee75476a6eb03f: Status 404 returned error can't find the container with id ed4a6834cffe87e109563d2fc29ac32dc9d5e79a172118d741ee75476a6eb03f Nov 03 18:12:53.984518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.984484 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:53.984643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.984562 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:53.984696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:53.984663 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:12:54.393652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.387848 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerStarted","Data":"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07"} Nov 03 18:12:54.393652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.387921 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerStarted","Data":"ed4a6834cffe87e109563d2fc29ac32dc9d5e79a172118d741ee75476a6eb03f"} Nov 03 18:12:54.426565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.424100 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:54.516971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.516933 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:54.517198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.516994 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:54.519779 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:54.519731 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podStartSLOduration=2.51971615 podStartE2EDuration="2.51971615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:12:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:12:54.515865617 +0000 UTC m=+2354.344949222" watchObservedRunningTime="2026-11-03 18:12:54.51971615 +0000 UTC m=+2354.348799751" Nov 03 18:12:55.430405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:55.430372 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:55.517458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:55.517420 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:55.517717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:55.517498 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:56.401239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.401205 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:56.402314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.402293 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:12:56.517863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.517325 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" start-of-body= Nov 03 18:12:56.517863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.517427 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.147:15021/healthz/ready\": dial tcp 10.132.0.147:15021: connect: connection refused" Nov 03 18:12:56.628441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.628411 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:12:56.659048 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659015 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659089 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659136 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659168 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659237 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659277 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659302 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659354 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659385 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\") pod \"6f9cc605-041d-43bc-b381-ae12b74e09a3\" (UID: \"6f9cc605-041d-43bc-b381-ae12b74e09a3\") " Nov 03 18:12:56.659782 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659651 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:56.659861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.659829 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:56.660399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660059 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:56.660399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660043 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:56.660399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660301 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:56.660643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660509 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.660643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660530 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6f9cc605-041d-43bc-b381-ae12b74e09a3-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.660643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660546 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.660643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660561 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.660643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.660594 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.662997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.662967 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:56.663426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.663394 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:56.663725 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.663698 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "kube-api-access-m5hkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:56.664127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.664112 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token" pod "6f9cc605-041d-43bc-b381-ae12b74e09a3" (UID: "6f9cc605-041d-43bc-b381-ae12b74e09a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:56.764520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.763967 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.764520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.764008 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hkq\" (UniqueName: \"kubernetes.io/projected/6f9cc605-041d-43bc-b381-ae12b74e09a3-kube-api-access-m5hkq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.764520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.764023 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:56.764520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:56.764038 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6f9cc605-041d-43bc-b381-ae12b74e09a3-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:57.410624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.410562 4980 generic.go:356] "Generic (PLEG): container finished" podID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerID="a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a" exitCode=0 Nov 03 18:12:57.410624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.410602 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerDied","Data":"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a"} Nov 03 18:12:57.410900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.410655 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" event={"ID":"6f9cc605-041d-43bc-b381-ae12b74e09a3","Type":"ContainerDied","Data":"f5d715e886a11e6ee50dc87504059fe068b3a6e5a2cf834f99952c260330210f"} Nov 03 18:12:57.410900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.410674 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6" Nov 03 18:12:57.410900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.410688 4980 scope.go:122] "RemoveContainer" containerID="a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a" Nov 03 18:12:57.432696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.432590 4980 scope.go:122] "RemoveContainer" containerID="6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194" Nov 03 18:12:57.442209 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.442191 4980 scope.go:122] "RemoveContainer" containerID="a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a" Nov 03 18:12:57.442486 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:57.442457 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a\": container with ID starting with a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a not found: ID does not exist" containerID="a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a" Nov 03 18:12:57.442616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.442492 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a"} err="failed to get container status \"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a\": rpc error: code = NotFound desc = could not find container \"a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a\": container with ID starting with a92df02878944fb0e5d76299a0e734fca4372fa056769d1107dff9fd12835d9a not found: ID does not exist" Nov 03 18:12:57.442616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.442511 4980 scope.go:122] "RemoveContainer" containerID="6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194" Nov 03 18:12:57.442868 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:57.442842 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194\": container with ID starting with 6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194 not found: ID does not exist" containerID="6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194" Nov 03 18:12:57.442920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.442869 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194"} err="failed to get container status \"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194\": rpc error: code = NotFound desc = could not find container \"6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194\": container with ID starting with 6af380ad2fa9c70aee24aed005a5d7f44dc388e53da8e178f4a042d8eecff194 not found: ID does not exist" Nov 03 18:12:57.522138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.522105 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:57.522645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.522488 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:57.523235 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.523217 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:12:57.641686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.641649 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:12:57.691473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:57.691438 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-2569c8-openshift-default-7b4f765579-7r6g6"] Nov 03 18:12:58.026119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.026092 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:12:58.082788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.082751 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.082983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.082857 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.082983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.082959 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083037 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083070 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083189 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:58.083270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083174 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083198 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:58.083374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083286 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083346 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083388 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\") pod \"f36b392c-adef-417f-883f-ee3d30774714\" (UID: \"f36b392c-adef-417f-883f-ee3d30774714\") " Nov 03 18:12:58.083476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083458 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:12:58.083654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083634 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:58.083805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.083784 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:58.084336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.084241 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.084336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.084268 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.084336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.084282 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.084336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.084298 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.084336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.084312 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36b392c-adef-417f-883f-ee3d30774714-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.085498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.085473 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:58.085938 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.085916 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:12:58.086066 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.086049 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "kube-api-access-mzs4c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:12:58.086147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.086131 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy" pod "f36b392c-adef-417f-883f-ee3d30774714" (UID: "f36b392c-adef-417f-883f-ee3d30774714"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:12:58.185662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.185625 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36b392c-adef-417f-883f-ee3d30774714-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.185662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.185663 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36b392c-adef-417f-883f-ee3d30774714-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.185874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.185678 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzs4c\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-kube-api-access-mzs4c\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.185874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.185691 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36b392c-adef-417f-883f-ee3d30774714-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:12:58.295934 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.295841 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" path="/var/lib/kubelet/pods/6f9cc605-041d-43bc-b381-ae12b74e09a3/volumes" Nov 03 18:12:58.420680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.420642 4980 generic.go:356] "Generic (PLEG): container finished" podID="f36b392c-adef-417f-883f-ee3d30774714" containerID="e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d" exitCode=0 Nov 03 18:12:58.420864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.420703 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" Nov 03 18:12:58.420864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.420713 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerDied","Data":"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d"} Nov 03 18:12:58.420864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.420782 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j" event={"ID":"f36b392c-adef-417f-883f-ee3d30774714","Type":"ContainerDied","Data":"76bbb87dd3a12f5c929dcddcd90a42728446445b0c57aac7121abace91b33fd9"} Nov 03 18:12:58.420864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.420812 4980 scope.go:122] "RemoveContainer" containerID="e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d" Nov 03 18:12:58.431636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.431611 4980 scope.go:122] "RemoveContainer" containerID="3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111" Nov 03 18:12:58.442393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.442374 4980 scope.go:122] "RemoveContainer" containerID="e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d" Nov 03 18:12:58.442746 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:58.442727 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d\": container with ID starting with e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d not found: ID does not exist" containerID="e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d" Nov 03 18:12:58.442818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.442758 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d"} err="failed to get container status \"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d\": rpc error: code = NotFound desc = could not find container \"e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d\": container with ID starting with e03a452e562f4ed79f3d7f718b1f3701181cb844721515153c28eaa192dd092d not found: ID does not exist" Nov 03 18:12:58.442818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.442778 4980 scope.go:122] "RemoveContainer" containerID="3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111" Nov 03 18:12:58.443027 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:12:58.443007 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111\": container with ID starting with 3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111 not found: ID does not exist" containerID="3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111" Nov 03 18:12:58.443091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.443040 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111"} err="failed to get container status \"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111\": rpc error: code = NotFound desc = could not find container \"3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111\": container with ID starting with 3cd2a90cd268259fcee84d7407d67cc8feaf2f25ffe762b2a97236e755504111 not found: ID does not exist" Nov 03 18:12:58.598986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.598892 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:12:58.605090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:12:58.605057 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-19d275-openshift-default-5c9978d69c-4mg7j"] Nov 03 18:13:00.297300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:00.297268 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36b392c-adef-417f-883f-ee3d30774714" path="/var/lib/kubelet/pods/f36b392c-adef-417f-883f-ee3d30774714/volumes" Nov 03 18:13:01.175503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.175467 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:13:01.176137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176120 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.176186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176139 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.176186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176163 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.176186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176168 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.176281 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176277 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.176315 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.176288 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.181396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.181378 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.241091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.241048 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:13:01.266778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.266731 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.266989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.266924 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf7mg\" (UniqueName: \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.295594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.291928 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-0bd8cf\"" Nov 03 18:13:01.295594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.292099 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-0bd8cf-dockercfg-g9chd\"" Nov 03 18:13:01.375634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.371807 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.375634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.372133 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf7mg\" (UniqueName: \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.428639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.425859 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.477600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.477536 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:13:01.479498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.478873 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.480072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.479890 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.480374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.480360 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.480474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.480460 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.480759 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.480746 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f9cc605-041d-43bc-b381-ae12b74e09a3" containerName="istio-proxy" Nov 03 18:13:01.480866 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.480856 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36b392c-adef-417f-883f-ee3d30774714" containerName="istio-proxy" Nov 03 18:13:01.487065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.485297 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:13:01.487065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.485404 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.488714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.488656 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf7mg\" (UniqueName: \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\") pod \"payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.491729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.490866 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 18:13:01.491729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.491083 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" containerID="cri-o://64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5" gracePeriod=30 Nov 03 18:13:01.494238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.494212 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:13:01.494444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.494398 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" containerID="cri-o://6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8" gracePeriod=30 Nov 03 18:13:01.497135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.496679 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:01.498220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.497870 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:13:01.498220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.498063 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" containerID="cri-o://7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c" gracePeriod=30 Nov 03 18:13:01.501003 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.500980 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:13:01.501369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.501176 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" containerID="cri-o://4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393" gracePeriod=30 Nov 03 18:13:01.583599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.580596 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhwlk\" (UniqueName: \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.583599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.580777 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.682068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.681968 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.682248 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.682197 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhwlk\" (UniqueName: \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.682718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.682696 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.745695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.745663 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhwlk\" (UniqueName: \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\") pod \"payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.861513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.861474 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:01.938312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:01.936352 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:13:01.941457 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:01.941420 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38533719_6e19_42e9_b8b7_cf6d00618c60.slice/crio-5d29b9fb917d5e7558f0ac7729c7964598b446bdb446f205175e4786ddd5f95d WatchSource:0}: Error finding container 5d29b9fb917d5e7558f0ac7729c7964598b446bdb446f205175e4786ddd5f95d: Status 404 returned error can't find the container with id 5d29b9fb917d5e7558f0ac7729c7964598b446bdb446f205175e4786ddd5f95d Nov 03 18:13:02.247108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.247077 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:13:02.248153 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:02.248123 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb4c043_bf27_488e_9007_6ca6cdc2034e.slice/crio-8d48b31f5bf608ed9d16f7d1ff2d3b2146940cdb1f68d7d7d3e8c62bf2f4c4e5 WatchSource:0}: Error finding container 8d48b31f5bf608ed9d16f7d1ff2d3b2146940cdb1f68d7d7d3e8c62bf2f4c4e5: Status 404 returned error can't find the container with id 8d48b31f5bf608ed9d16f7d1ff2d3b2146940cdb1f68d7d7d3e8c62bf2f4c4e5 Nov 03 18:13:02.495900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.495792 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" event={"ID":"38533719-6e19-42e9-b8b7-cf6d00618c60","Type":"ContainerStarted","Data":"00cb819a60e7bd4a6b4af8404d187a74f2d70074118b3f2ff66bc67299736da8"} Nov 03 18:13:02.495900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.495868 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" event={"ID":"38533719-6e19-42e9-b8b7-cf6d00618c60","Type":"ContainerStarted","Data":"5d29b9fb917d5e7558f0ac7729c7964598b446bdb446f205175e4786ddd5f95d"} Nov 03 18:13:02.496332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.495905 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:02.497946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.497918 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" event={"ID":"3bb4c043-bf27-488e-9007-6ca6cdc2034e","Type":"ContainerStarted","Data":"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d"} Nov 03 18:13:02.498069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.497962 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" event={"ID":"3bb4c043-bf27-488e-9007-6ca6cdc2034e","Type":"ContainerStarted","Data":"8d48b31f5bf608ed9d16f7d1ff2d3b2146940cdb1f68d7d7d3e8c62bf2f4c4e5"} Nov 03 18:13:02.498199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.498185 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:02.615035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:02.614978 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" podStartSLOduration=1.6149623709999998 podStartE2EDuration="1.614962371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:02.614044412 +0000 UTC m=+2362.443128017" watchObservedRunningTime="2026-11-03 18:13:02.614962371 +0000 UTC m=+2362.444045972" Nov 03 18:13:05.190799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:05.190762 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-sg8bl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:13:05.191322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:05.190819 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:13:06.801975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.801939 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:13:06.821499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.821468 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:13:06.831847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.831823 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 18:13:06.969982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.969950 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.969994 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970030 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.970166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970055 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970095 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970133 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970174 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970238 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970269 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970283 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970335 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970368 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970419 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970447 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970497 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970522 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970616 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970661 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970724 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970725 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970751 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970790 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970801 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.970948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.970878 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971030 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971042 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971061 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971111 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\") pod \"71271103-783a-4333-97a1-c8a25dba560a\" (UID: \"71271103-783a-4333-97a1-c8a25dba560a\") " Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971964 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971176 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972305 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971186 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972016 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972264 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.971474 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:06.972486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972348 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.973181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972598 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.973181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972653 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\") pod \"57a4e16c-3777-4d4a-8c05-15c543064878\" (UID: \"57a4e16c-3777-4d4a-8c05-15c543064878\") " Nov 03 18:13:06.973181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.972682 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\") pod \"66cc9f76-7e43-45ce-9998-7a6fbf224073\" (UID: \"66cc9f76-7e43-45ce-9998-7a6fbf224073\") " Nov 03 18:13:06.973181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973136 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.973181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973173 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.973444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973264 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:06.973505 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973451 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:06.973733 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973711 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973742 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973757 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71271103-783a-4333-97a1-c8a25dba560a-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973771 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973785 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973798 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.973817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973811 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973826 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/57a4e16c-3777-4d4a-8c05-15c543064878-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973841 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973854 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973867 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973879 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/66cc9f76-7e43-45ce-9998-7a6fbf224073-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973895 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973909 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/57a4e16c-3777-4d4a-8c05-15c543064878-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973922 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.973936 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.974059 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "kube-api-access-phh6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.974126 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.974104 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:06.975292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.975082 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.976221 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.976195 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.976449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.976415 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.976449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.976456 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "kube-api-access-bfsm9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.977838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.976811 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.977838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.977056 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.977838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.977541 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo" pod "71271103-783a-4333-97a1-c8a25dba560a" (UID: "71271103-783a-4333-97a1-c8a25dba560a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:06.977838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.977543 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy" pod "57a4e16c-3777-4d4a-8c05-15c543064878" (UID: "57a4e16c-3777-4d4a-8c05-15c543064878"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:06.978644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.978565 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz" pod "66cc9f76-7e43-45ce-9998-7a6fbf224073" (UID: "66cc9f76-7e43-45ce-9998-7a6fbf224073"). InnerVolumeSpecName "kube-api-access-f6phz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:06.980503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:06.980464 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" podStartSLOduration=5.9804516549999995 podStartE2EDuration="5.980451655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:02.750513625 +0000 UTC m=+2362.579597229" watchObservedRunningTime="2026-11-03 18:13:06.980451655 +0000 UTC m=+2366.809535256" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075172 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/57a4e16c-3777-4d4a-8c05-15c543064878-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075206 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075220 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71271103-783a-4333-97a1-c8a25dba560a-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075234 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075248 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfsm9\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-kube-api-access-bfsm9\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075265 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6phz\" (UniqueName: \"kubernetes.io/projected/66cc9f76-7e43-45ce-9998-7a6fbf224073-kube-api-access-f6phz\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075277 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71271103-783a-4333-97a1-c8a25dba560a-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075290 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-phh6w\" (UniqueName: \"kubernetes.io/projected/71271103-783a-4333-97a1-c8a25dba560a-kube-api-access-phh6w\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075302 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075313 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/57a4e16c-3777-4d4a-8c05-15c543064878-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.075600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.075324 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/66cc9f76-7e43-45ce-9998-7a6fbf224073-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.079964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.079934 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4wzlm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" start-of-body= Nov 03 18:13:07.080107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.080009 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.107:15021/healthz/ready\": dial tcp 10.132.0.107:15021: connect: connection refused" Nov 03 18:13:07.168233 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.168201 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:13:07.276886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.276850 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.277080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.276957 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.277080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277013 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.277080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277058 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.277244 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277186 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.277328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277296 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:07.277441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277415 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:07.277758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.277732 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.278042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278021 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.278250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278231 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.278536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278491 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:07.278693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278533 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\") pod \"a240db05-8056-412e-baeb-a22cea0e066c\" (UID: \"a240db05-8056-412e-baeb-a22cea0e066c\") " Nov 03 18:13:07.278693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278554 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:07.278948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.278915 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:07.279195 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279168 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a240db05-8056-412e-baeb-a22cea0e066c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.279278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279194 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.279278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279210 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.279278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279225 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.279278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279241 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.279992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.279968 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:07.280367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.280341 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "kube-api-access-bnkxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:07.281121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.281100 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:07.281491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.281470 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token" pod "a240db05-8056-412e-baeb-a22cea0e066c" (UID: "a240db05-8056-412e-baeb-a22cea0e066c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:07.380968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.380834 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnkxx\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-kube-api-access-bnkxx\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.380968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.380879 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a240db05-8056-412e-baeb-a22cea0e066c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.380968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.380895 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a240db05-8056-412e-baeb-a22cea0e066c-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.380968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.380909 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a240db05-8056-412e-baeb-a22cea0e066c-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:07.533558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.533523 4980 generic.go:356] "Generic (PLEG): container finished" podID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerID="64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5" exitCode=0 Nov 03 18:13:07.533763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.533592 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" Nov 03 18:13:07.533763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.533606 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" event={"ID":"66cc9f76-7e43-45ce-9998-7a6fbf224073","Type":"ContainerDied","Data":"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5"} Nov 03 18:13:07.533763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.533666 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl" event={"ID":"66cc9f76-7e43-45ce-9998-7a6fbf224073","Type":"ContainerDied","Data":"fe1c483082e1e1dcce86b147324586aaeb55b6fcefb0c5c62273a15d2c936868"} Nov 03 18:13:07.533763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.533695 4980 scope.go:122] "RemoveContainer" containerID="64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5" Nov 03 18:13:07.535282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.535258 4980 generic.go:356] "Generic (PLEG): container finished" podID="71271103-783a-4333-97a1-c8a25dba560a" containerID="4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393" exitCode=0 Nov 03 18:13:07.535401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.535369 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" event={"ID":"71271103-783a-4333-97a1-c8a25dba560a","Type":"ContainerDied","Data":"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393"} Nov 03 18:13:07.535483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.535413 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" event={"ID":"71271103-783a-4333-97a1-c8a25dba560a","Type":"ContainerDied","Data":"9933b6691bb1d634b6cc7face2a13542f92b9c829244b8c8cd1d0a509a27781c"} Nov 03 18:13:07.535563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.535483 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn" Nov 03 18:13:07.538531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.538506 4980 generic.go:356] "Generic (PLEG): container finished" podID="a240db05-8056-412e-baeb-a22cea0e066c" containerID="6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8" exitCode=0 Nov 03 18:13:07.538764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.538579 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" Nov 03 18:13:07.538764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.538603 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" event={"ID":"a240db05-8056-412e-baeb-a22cea0e066c","Type":"ContainerDied","Data":"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8"} Nov 03 18:13:07.538764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.538654 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm" event={"ID":"a240db05-8056-412e-baeb-a22cea0e066c","Type":"ContainerDied","Data":"e0aef5bb7b1188682f902897d667153b82479dcbfc8b1257ca8758b78a55e2b0"} Nov 03 18:13:07.540377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.540354 4980 generic.go:356] "Generic (PLEG): container finished" podID="57a4e16c-3777-4d4a-8c05-15c543064878" containerID="7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c" exitCode=0 Nov 03 18:13:07.540488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.540420 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" event={"ID":"57a4e16c-3777-4d4a-8c05-15c543064878","Type":"ContainerDied","Data":"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c"} Nov 03 18:13:07.540488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.540461 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" event={"ID":"57a4e16c-3777-4d4a-8c05-15c543064878","Type":"ContainerDied","Data":"254c607336805010b63dd25745e30be24cf7fd3d6728af5f7cbe94e6845078c9"} Nov 03 18:13:07.540692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.540530 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9" Nov 03 18:13:07.550645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.550620 4980 scope.go:122] "RemoveContainer" containerID="64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5" Nov 03 18:13:07.551014 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:07.550986 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5\": container with ID starting with 64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5 not found: ID does not exist" containerID="64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5" Nov 03 18:13:07.551089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.551031 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5"} err="failed to get container status \"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5\": rpc error: code = NotFound desc = could not find container \"64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5\": container with ID starting with 64231637beb768a435a437cc0716b8a847c5d3266151ce450d57b065a09663f5 not found: ID does not exist" Nov 03 18:13:07.551089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.551055 4980 scope.go:122] "RemoveContainer" containerID="4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393" Nov 03 18:13:07.561334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.561312 4980 scope.go:122] "RemoveContainer" containerID="4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393" Nov 03 18:13:07.561655 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:07.561633 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393\": container with ID starting with 4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393 not found: ID does not exist" containerID="4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393" Nov 03 18:13:07.561740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.561667 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393"} err="failed to get container status \"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393\": rpc error: code = NotFound desc = could not find container \"4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393\": container with ID starting with 4db0b9dd082a5f13a60dde549547a6012c723a09bbde87c2fe443fcd3234c393 not found: ID does not exist" Nov 03 18:13:07.561740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.561694 4980 scope.go:122] "RemoveContainer" containerID="6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8" Nov 03 18:13:07.571921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.571901 4980 scope.go:122] "RemoveContainer" containerID="6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8" Nov 03 18:13:07.572230 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:07.572211 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8\": container with ID starting with 6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8 not found: ID does not exist" containerID="6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8" Nov 03 18:13:07.572295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.572239 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8"} err="failed to get container status \"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8\": rpc error: code = NotFound desc = could not find container \"6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8\": container with ID starting with 6521da689927e4396c9c84a32bf4db21942904538db1b223c978fb85925113b8 not found: ID does not exist" Nov 03 18:13:07.572295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.572257 4980 scope.go:122] "RemoveContainer" containerID="7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c" Nov 03 18:13:07.581793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.581772 4980 scope.go:122] "RemoveContainer" containerID="7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c" Nov 03 18:13:07.582097 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:07.582079 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c\": container with ID starting with 7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c not found: ID does not exist" containerID="7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c" Nov 03 18:13:07.582158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.582106 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c"} err="failed to get container status \"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c\": rpc error: code = NotFound desc = could not find container \"7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c\": container with ID starting with 7e884186fa5ad2bfc8146551f2ac8218f298137878484e91f49d85d4f67b181c not found: ID does not exist" Nov 03 18:13:07.704527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.704492 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 18:13:07.712564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.712536 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-sg8bl"] Nov 03 18:13:07.827204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.827168 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:13:07.833086 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.833054 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-nm6mn"] Nov 03 18:13:07.946583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.946533 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:13:07.952525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:07.952499 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4wzlm"] Nov 03 18:13:08.065715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.065630 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:13:08.071888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.071858 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-2ggk9"] Nov 03 18:13:08.293939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.293899 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" path="/var/lib/kubelet/pods/57a4e16c-3777-4d4a-8c05-15c543064878/volumes" Nov 03 18:13:08.294349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.294335 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" path="/var/lib/kubelet/pods/66cc9f76-7e43-45ce-9998-7a6fbf224073/volumes" Nov 03 18:13:08.294840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.294825 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71271103-783a-4333-97a1-c8a25dba560a" path="/var/lib/kubelet/pods/71271103-783a-4333-97a1-c8a25dba560a/volumes" Nov 03 18:13:08.295612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:08.295553 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a240db05-8056-412e-baeb-a22cea0e066c" path="/var/lib/kubelet/pods/a240db05-8056-412e-baeb-a22cea0e066c/volumes" Nov 03 18:13:10.741708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:10.741673 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:13:10.742228 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:10.741940 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" containerID="cri-o://6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67" gracePeriod=30 Nov 03 18:13:13.508271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:13.508239 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:13:13.508749 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:13.508336 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:13:14.520901 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:14.520865 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:13:16.023513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.023454 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:13:16.129876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.129841 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130005 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130101 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130161 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130194 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:16.130232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130204 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130269 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130312 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130433 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130461 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\") pod \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\" (UID: \"51d23b7b-15df-4aa6-ae18-2e222ce7316c\") " Nov 03 18:13:16.130658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.130617 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:16.131094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131000 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:16.131515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131192 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:16.131515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131486 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:16.131844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131648 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.131844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131685 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.131844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131701 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.131844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.131716 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.132808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.132784 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:16.132966 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.132946 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:16.133075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.133052 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "kube-api-access-rqzqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:16.133482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.133458 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo" pod "51d23b7b-15df-4aa6-ae18-2e222ce7316c" (UID: "51d23b7b-15df-4aa6-ae18-2e222ce7316c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:16.290699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.246165 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.290699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.246203 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.290699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.246218 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.290699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.246231 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/51d23b7b-15df-4aa6-ae18-2e222ce7316c-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.290699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.246249 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqzqt\" (UniqueName: \"kubernetes.io/projected/51d23b7b-15df-4aa6-ae18-2e222ce7316c-kube-api-access-rqzqt\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:16.336791 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.336749 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 18:13:16.613122 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.613026 4980 generic.go:356] "Generic (PLEG): container finished" podID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerID="6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67" exitCode=0 Nov 03 18:13:16.613122 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.613085 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" Nov 03 18:13:16.613357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.613105 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerDied","Data":"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67"} Nov 03 18:13:16.613357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.613185 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82" event={"ID":"51d23b7b-15df-4aa6-ae18-2e222ce7316c","Type":"ContainerDied","Data":"2cd78c0dad6b96d4666ac1d17fada7aedd08088e00575f405a7f2ec54da0a57d"} Nov 03 18:13:16.613357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.613216 4980 scope.go:122] "RemoveContainer" containerID="6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67" Nov 03 18:13:16.623752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.623728 4980 scope.go:122] "RemoveContainer" containerID="e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e" Nov 03 18:13:16.633552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.633532 4980 scope.go:122] "RemoveContainer" containerID="6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67" Nov 03 18:13:16.633860 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:16.633836 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67\": container with ID starting with 6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67 not found: ID does not exist" containerID="6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67" Nov 03 18:13:16.633943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.633871 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67"} err="failed to get container status \"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67\": rpc error: code = NotFound desc = could not find container \"6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67\": container with ID starting with 6e243c132667f97026181c548a3fd1a21ba9619b7ec8ccf6e750ef296d21ba67 not found: ID does not exist" Nov 03 18:13:16.633943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.633898 4980 scope.go:122] "RemoveContainer" containerID="e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e" Nov 03 18:13:16.634168 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:16.634150 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e\": container with ID starting with e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e not found: ID does not exist" containerID="e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e" Nov 03 18:13:16.634215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.634177 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e"} err="failed to get container status \"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e\": rpc error: code = NotFound desc = could not find container \"e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e\": container with ID starting with e88e6cbea9680e46b483b6ee690ad041c0c79319de906676b790ff0c57c7740e not found: ID does not exist" Nov 03 18:13:16.785449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.785403 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:13:16.792840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.792812 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-62aead-openshift-default-6d4f8b4c75-66c82"] Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.802783 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804070 4980 cpu_manager.go:414] "containerMap: removing container" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804092 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804139 4980 cpu_manager.go:414] "containerMap: removing container" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804148 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804230 4980 cpu_manager.go:414] "containerMap: removing container" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804238 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804262 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804270 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804289 4980 cpu_manager.go:414] "containerMap: removing container" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804296 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804338 4980 cpu_manager.go:414] "containerMap: removing container" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804345 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804617 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a4e16c-3777-4d4a-8c05-15c543064878" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804636 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="66cc9f76-7e43-45ce-9998-7a6fbf224073" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804659 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804686 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804703 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a240db05-8056-412e-baeb-a22cea0e066c" containerName="istio-proxy" Nov 03 18:13:16.805611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.804713 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="71271103-783a-4333-97a1-c8a25dba560a" containerName="istio-proxy" Nov 03 18:13:16.811202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.811170 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.853972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.853934 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.854142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.854036 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.854238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.854182 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.854290 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.854233 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfdmm\" (UniqueName: \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.866682 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.866606 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ca6b33-serving-cert\"" Nov 03 18:13:16.868703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.868678 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:13:16.955311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.955278 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.955487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.955329 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfdmm\" (UniqueName: \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.955487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.955417 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.955487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.955470 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.955678 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:16.955633 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-serving-cert: secret "maas-api-e2e-inf-a-ca6b33-serving-cert" not found Nov 03 18:13:16.955747 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:16.955704 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls podName:4ad45e40-7c86-4b9e-919c-8b742871defc nodeName:}" failed. No retries permitted until 2026-11-03 18:13:17.455684352 +0000 UTC m=+2377.284767948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls") pod "maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" (UID: "4ad45e40-7c86-4b9e-919c-8b742871defc") : secret "maas-api-e2e-inf-a-ca6b33-serving-cert" not found Nov 03 18:13:16.956220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.956194 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:16.958056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:16.958035 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:17.020641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:17.020595 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfdmm\" (UniqueName: \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:17.470099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:17.470067 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:17.472712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:17.472686 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") pod \"maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:17.728438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:17.728343 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:18.116121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:18.116091 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:13:18.118581 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:18.118531 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ad45e40_7c86_4b9e_919c_8b742871defc.slice/crio-5e0f3eb1eec0c4196a809b2a6cb038ac47df93c2db6b9cbbc085634a55a4961b WatchSource:0}: Error finding container 5e0f3eb1eec0c4196a809b2a6cb038ac47df93c2db6b9cbbc085634a55a4961b: Status 404 returned error can't find the container with id 5e0f3eb1eec0c4196a809b2a6cb038ac47df93c2db6b9cbbc085634a55a4961b Nov 03 18:13:18.120368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:18.120339 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:13:18.295340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:18.295299 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51d23b7b-15df-4aa6-ae18-2e222ce7316c" path="/var/lib/kubelet/pods/51d23b7b-15df-4aa6-ae18-2e222ce7316c/volumes" Nov 03 18:13:18.629219 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:18.629179 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" event={"ID":"4ad45e40-7c86-4b9e-919c-8b742871defc","Type":"ContainerStarted","Data":"5e0f3eb1eec0c4196a809b2a6cb038ac47df93c2db6b9cbbc085634a55a4961b"} Nov 03 18:13:19.637647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:19.637609 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" event={"ID":"4ad45e40-7c86-4b9e-919c-8b742871defc","Type":"ContainerStarted","Data":"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef"} Nov 03 18:13:19.638048 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:19.637677 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:19.775293 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:19.775236 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" podStartSLOduration=2.968222013 podStartE2EDuration="3.775220902s" totalImagesPullingTime="806.998889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:19.77366171 +0000 UTC m=+2379.602745311" watchObservedRunningTime="2026-11-03 18:13:19.775220902 +0000 UTC m=+2379.604304504" Nov 03 18:13:24.741555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.741465 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7198-dd44d6"] Nov 03 18:13:24.748246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.748210 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:24.803858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.803824 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-dd44d6"] Nov 03 18:13:24.849318 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.849290 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-6hkxc\"" Nov 03 18:13:24.874988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.874952 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrfn\" (UniqueName: \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\") pod \"test-curl-7198-dd44d6\" (UID: \"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3\") " pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:24.976019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:24.975987 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrfn\" (UniqueName: \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\") pod \"test-curl-7198-dd44d6\" (UID: \"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3\") " pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:25.038981 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:25.038910 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrfn\" (UniqueName: \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\") pod \"test-curl-7198-dd44d6\" (UID: \"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3\") " pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:25.062872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:25.062844 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:25.441300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:25.441275 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7198-dd44d6"] Nov 03 18:13:25.443468 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:25.443440 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5c5737a_616e_42c4_9ba8_b0fcfdb814a3.slice/crio-42a921f2f6df69d09758de612eead08ce73692e2fbf982e8134ebade7e3704f0 WatchSource:0}: Error finding container 42a921f2f6df69d09758de612eead08ce73692e2fbf982e8134ebade7e3704f0: Status 404 returned error can't find the container with id 42a921f2f6df69d09758de612eead08ce73692e2fbf982e8134ebade7e3704f0 Nov 03 18:13:25.650914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:25.650886 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:13:25.678266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:25.678231 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-dd44d6" event={"ID":"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3","Type":"ContainerStarted","Data":"42a921f2f6df69d09758de612eead08ce73692e2fbf982e8134ebade7e3704f0"} Nov 03 18:13:26.686374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:26.686330 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7198-dd44d6" event={"ID":"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3","Type":"ContainerStarted","Data":"4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0"} Nov 03 18:13:26.803261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:26.803157 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7198-dd44d6" podStartSLOduration=1.731092764 podStartE2EDuration="2.803141719s" totalImagesPullingTime="1.072048955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:26.801873616 +0000 UTC m=+2386.630957217" watchObservedRunningTime="2026-11-03 18:13:26.803141719 +0000 UTC m=+2386.632225321" Nov 03 18:13:27.219359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:27.219324 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 18:13:28.418980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:28.418921 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7198-dd44d6"] Nov 03 18:13:28.422713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:28.422684 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7198-dd44d6"] Nov 03 18:13:28.720093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:28.720035 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7198-dd44d6" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" containerName="test-curl-7198-dd44d6" containerID="cri-o://4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0" gracePeriod=2 Nov 03 18:13:28.771280 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:28.771241 4980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-dd44d6\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-1.compute.internal' and this object" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:30.341326 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:30.341285 4980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-dd44d6\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-1.compute.internal' and this object" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:30.975788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:30.975728 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:31.071515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.071483 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\" (UniqueName: \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\") pod \"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3\" (UID: \"f5c5737a-616e-42c4-9ba8-b0fcfdb814a3\") " Nov 03 18:13:31.073774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.073749 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn" pod "f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" (UID: "f5c5737a-616e-42c4-9ba8-b0fcfdb814a3"). InnerVolumeSpecName "kube-api-access-skrfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:31.172939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.172904 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrfn\" (UniqueName: \"kubernetes.io/projected/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3-kube-api-access-skrfn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.382626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.382490 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" containerID="cri-o://c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978" gracePeriod=45 Nov 03 18:13:31.648246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.648215 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:13:31.742241 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.742211 4980 generic.go:356] "Generic (PLEG): container finished" podID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerID="c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978" exitCode=0 Nov 03 18:13:31.742408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.742316 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" event={"ID":"c296cbf0-a68b-41db-b467-57a28081f9f1","Type":"ContainerDied","Data":"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978"} Nov 03 18:13:31.742408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.742326 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" Nov 03 18:13:31.742408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.742371 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7" event={"ID":"c296cbf0-a68b-41db-b467-57a28081f9f1","Type":"ContainerDied","Data":"07f6da192503675047075bce48162c1a00b4ecb1c299e3fd5479f07aa557a5b3"} Nov 03 18:13:31.742559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.742399 4980 scope.go:122] "RemoveContainer" containerID="c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978" Nov 03 18:13:31.743991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.743952 4980 generic.go:356] "Generic (PLEG): container finished" podID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" containerID="4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0" exitCode=137 Nov 03 18:13:31.744111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.743993 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:31.754239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.754222 4980 scope.go:122] "RemoveContainer" containerID="c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978" Nov 03 18:13:31.754583 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:31.754548 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978\": container with ID starting with c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978 not found: ID does not exist" containerID="c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978" Nov 03 18:13:31.754648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.754597 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978"} err="failed to get container status \"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978\": rpc error: code = NotFound desc = could not find container \"c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978\": container with ID starting with c4fe31ee19bc89cf753547b44980d1d5fb23a1955fc4d95c97a02391fed77978 not found: ID does not exist" Nov 03 18:13:31.754648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.754620 4980 scope.go:122] "RemoveContainer" containerID="4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0" Nov 03 18:13:31.765657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.765631 4980 scope.go:122] "RemoveContainer" containerID="4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0" Nov 03 18:13:31.766022 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:31.765998 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0\": container with ID starting with 4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0 not found: ID does not exist" containerID="4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0" Nov 03 18:13:31.766129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.766030 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0"} err="failed to get container status \"4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0\": rpc error: code = NotFound desc = could not find container \"4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0\": container with ID starting with 4ac11380e2949d17491a11fb45417836bda0b2a5db69f4215d9a9b4c9d19ceb0 not found: ID does not exist" Nov 03 18:13:31.779510 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779478 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.779714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779563 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.779787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779732 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\" (UniqueName: \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.779787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779767 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.779898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779871 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\" (UniqueName: \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.779961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.779908 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\") pod \"c296cbf0-a68b-41db-b467-57a28081f9f1\" (UID: \"c296cbf0-a68b-41db-b467-57a28081f9f1\") " Nov 03 18:13:31.781065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.780884 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:31.781253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.781233 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:31.781617 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.781543 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:31.783498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.783473 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:13:31.783498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.783474 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:31.783919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.783901 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp" pod "c296cbf0-a68b-41db-b467-57a28081f9f1" (UID: "c296cbf0-a68b-41db-b467-57a28081f9f1"). InnerVolumeSpecName "kube-api-access-xd6sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:31.874084 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:31.874043 4980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-dd44d6\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-1.compute.internal' and this object" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:31.882121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882086 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.882121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882118 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.882121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882128 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c296cbf0-a68b-41db-b467-57a28081f9f1-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.882373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882137 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.882373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882146 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd6sp\" (UniqueName: \"kubernetes.io/projected/c296cbf0-a68b-41db-b467-57a28081f9f1-kube-api-access-xd6sp\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:31.882373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:31.882155 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c296cbf0-a68b-41db-b467-57a28081f9f1-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:32.103243 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:32.103206 4980 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7198-dd44d6\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-234.us-west-1.compute.internal' and this object" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" pod="opendatahub/test-curl-7198-dd44d6" Nov 03 18:13:32.218017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:32.217984 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 18:13:32.220253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:32.220210 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" containerID="cri-o://c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964" gracePeriod=25 Nov 03 18:13:32.223847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:32.223820 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-07f297/distinct-2-w0-07f297-kserve-7b5fbff9-xm8n7"] Nov 03 18:13:32.294184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:32.294151 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" path="/var/lib/kubelet/pods/c296cbf0-a68b-41db-b467-57a28081f9f1/volumes" Nov 03 18:13:32.294545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:32.294531 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" path="/var/lib/kubelet/pods/f5c5737a-616e-42c4-9ba8-b0fcfdb814a3/volumes" Nov 03 18:13:33.691384 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:33.691345 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:13:41.050734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:41.050701 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:13:41.054840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:41.054814 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:13:41.055603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:41.055583 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:13:41.059904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:41.059884 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:13:43.691084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:43.691042 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:13:44.176511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:44.176426 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:13:44.176699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:44.176670 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" containerID="cri-o://db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49" gracePeriod=30 Nov 03 18:13:44.521817 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:44.521780 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:13:45.179234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.179193 4980 patch_prober.go:29] interesting pod/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:13:45.179722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.179269 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:13:45.672032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.671992 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:13:45.672599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672583 4980 cpu_manager.go:414] "containerMap: removing container" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" Nov 03 18:13:45.672666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672600 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" Nov 03 18:13:45.672666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672629 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" containerName="test-curl-7198-dd44d6" Nov 03 18:13:45.672666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672635 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" containerName="test-curl-7198-dd44d6" Nov 03 18:13:45.672763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672750 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c296cbf0-a68b-41db-b467-57a28081f9f1" containerName="main" Nov 03 18:13:45.672797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.672763 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5c5737a-616e-42c4-9ba8-b0fcfdb814a3" containerName="test-curl-7198-dd44d6" Nov 03 18:13:45.677532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.677514 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.821657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.811026 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-43bdc0-openshift-default-dockercfg-cxx26\"" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825033 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825093 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825124 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825242 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp7hn\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825302 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825349 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825402 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825435 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.841803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.825487 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.895692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.893586 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:13:45.926396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926299 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp7hn\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.926396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926387 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.926695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926444 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.926695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926494 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.926695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926524 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.926915 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927021 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927022 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927061 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927066 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927108 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927165 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927504 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.927702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.927684 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.929604 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.929542 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.929815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.929797 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.989523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.989485 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:45.989807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:45.989784 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp7hn\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\") pod \"e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:46.005316 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.005289 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:46.443428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.443398 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:13:46.445369 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:46.445343 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod759e0b50_5907_469c_b9b0_c6b1aca03260.slice/crio-49e0aba6f89be1af7c5cd2c8467f91b96447ab732682d81681e95c76bc32b166 WatchSource:0}: Error finding container 49e0aba6f89be1af7c5cd2c8467f91b96447ab732682d81681e95c76bc32b166: Status 404 returned error can't find the container with id 49e0aba6f89be1af7c5cd2c8467f91b96447ab732682d81681e95c76bc32b166 Nov 03 18:13:46.448000 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.447965 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:13:46.448095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.448039 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:13:46.448095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.448088 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:13:46.913744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.913621 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerStarted","Data":"d73a72c40c516cf032c174cc31cf35b175baa6344a13972c6618b307d16d29d2"} Nov 03 18:13:46.913744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:46.913712 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerStarted","Data":"49e0aba6f89be1af7c5cd2c8467f91b96447ab732682d81681e95c76bc32b166"} Nov 03 18:13:47.005418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:47.005386 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:47.011524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:47.011499 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:47.042533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:47.042477 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podStartSLOduration=2.042463151 podStartE2EDuration="2.042463151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:47.039652977 +0000 UTC m=+2406.868736579" watchObservedRunningTime="2026-11-03 18:13:47.042463151 +0000 UTC m=+2406.871546818" Nov 03 18:13:47.919972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:47.919946 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:47.920886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:47.920865 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:13:49.457769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.457713 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:13:49.583555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583523 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583678 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583721 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583771 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583814 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583844 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583941 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.583994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.583987 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.584098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584043 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\") pod \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\" (UID: \"40a4fcb1-8f46-4bcd-8538-9c7734b42c73\") " Nov 03 18:13:49.584306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584282 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584345 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584516 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584681 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584943 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584965 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584977 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.585045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.584992 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.585504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.585099 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.586204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.586173 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:49.586705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.586677 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "kube-api-access-9qkm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:49.587058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.587040 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:49.587124 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.587046 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy" pod "40a4fcb1-8f46-4bcd-8538-9c7734b42c73" (UID: "40a4fcb1-8f46-4bcd-8538-9c7734b42c73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:49.686096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.686064 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.686096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.686092 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.686096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.686102 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.686351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.686111 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qkm7\" (UniqueName: \"kubernetes.io/projected/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-kube-api-access-9qkm7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.686351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.686122 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/40a4fcb1-8f46-4bcd-8538-9c7734b42c73-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:49.934111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.934017 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" Nov 03 18:13:49.934111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.933946 4980 generic.go:356] "Generic (PLEG): container finished" podID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerID="db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49" exitCode=0 Nov 03 18:13:49.934366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.934006 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerDied","Data":"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49"} Nov 03 18:13:49.934366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.934226 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs" event={"ID":"40a4fcb1-8f46-4bcd-8538-9c7734b42c73","Type":"ContainerDied","Data":"6621b76bb896874c16390b510924fef134458c85baf84e3bd460f4d66adb5637"} Nov 03 18:13:49.934366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.934261 4980 scope.go:122] "RemoveContainer" containerID="db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49" Nov 03 18:13:49.945512 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.945493 4980 scope.go:122] "RemoveContainer" containerID="b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968" Nov 03 18:13:49.959018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.958998 4980 scope.go:122] "RemoveContainer" containerID="db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49" Nov 03 18:13:49.959313 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:49.959290 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49\": container with ID starting with db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49 not found: ID does not exist" containerID="db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49" Nov 03 18:13:49.959367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.959326 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49"} err="failed to get container status \"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49\": rpc error: code = NotFound desc = could not find container \"db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49\": container with ID starting with db3240dd23d7aff9db9b0ed963d4a3e0193d7fad50375a0bc34f2fe284e1ca49 not found: ID does not exist" Nov 03 18:13:49.959367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.959348 4980 scope.go:122] "RemoveContainer" containerID="b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968" Nov 03 18:13:49.959646 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:49.959617 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968\": container with ID starting with b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968 not found: ID does not exist" containerID="b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968" Nov 03 18:13:49.959729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:49.959656 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968"} err="failed to get container status \"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968\": rpc error: code = NotFound desc = could not find container \"b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968\": container with ID starting with b42d3c0ed824597d4bc039b1a56930a9662feb8ad5580a309171c2a3c4650968 not found: ID does not exist" Nov 03 18:13:50.108195 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:50.108165 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:13:50.158467 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:50.158432 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-528e53-openshift-default-759b55894f-jnxvs"] Nov 03 18:13:50.184622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:50.184518 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:13:50.184805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:50.184774 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" containerID="cri-o://04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7" gracePeriod=30 Nov 03 18:13:50.397596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:50.394065 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" path="/var/lib/kubelet/pods/40a4fcb1-8f46-4bcd-8538-9c7734b42c73/volumes" Nov 03 18:13:52.512810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.512777 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 18:13:52.570344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.570307 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\") pod \"e7bc4f39-5ff4-411d-85cf-a248ec146123\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " Nov 03 18:13:52.570513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.570501 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\" (UniqueName: \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\") pod \"e7bc4f39-5ff4-411d-85cf-a248ec146123\" (UID: \"e7bc4f39-5ff4-411d-85cf-a248ec146123\") " Nov 03 18:13:52.570819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.570791 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume" pod "e7bc4f39-5ff4-411d-85cf-a248ec146123" (UID: "e7bc4f39-5ff4-411d-85cf-a248ec146123"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:52.570956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.570937 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7bc4f39-5ff4-411d-85cf-a248ec146123-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:52.572948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.572920 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn" pod "e7bc4f39-5ff4-411d-85cf-a248ec146123" (UID: "e7bc4f39-5ff4-411d-85cf-a248ec146123"). InnerVolumeSpecName "kube-api-access-lzpjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:52.672385 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.672343 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzpjn\" (UniqueName: \"kubernetes.io/projected/e7bc4f39-5ff4-411d-85cf-a248ec146123-kube-api-access-lzpjn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:52.993290 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.993257 4980 generic.go:356] "Generic (PLEG): container finished" podID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerID="c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964" exitCode=0 Nov 03 18:13:52.993511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.993320 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" Nov 03 18:13:52.993511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.993335 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" event={"ID":"e7bc4f39-5ff4-411d-85cf-a248ec146123","Type":"ContainerDied","Data":"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964"} Nov 03 18:13:52.993511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.993396 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r" event={"ID":"e7bc4f39-5ff4-411d-85cf-a248ec146123","Type":"ContainerDied","Data":"4e973696922cc0100c1f603b280668ee255a37fed2133e55cd69c3082982c812"} Nov 03 18:13:52.993511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:52.993422 4980 scope.go:122] "RemoveContainer" containerID="c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964" Nov 03 18:13:53.006693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:53.006562 4980 scope.go:122] "RemoveContainer" containerID="c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964" Nov 03 18:13:53.007212 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:53.007179 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964\": container with ID starting with c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964 not found: ID does not exist" containerID="c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964" Nov 03 18:13:53.007349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:53.007222 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964"} err="failed to get container status \"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964\": rpc error: code = NotFound desc = could not find container \"c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964\": container with ID starting with c511d05f2bfa3d692ac3eafb20092163a1bb9417150a461da00f1c5337ae8964 not found: ID does not exist" Nov 03 18:13:53.164325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:53.164283 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 18:13:53.169906 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:53.169879 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w0-07f297-5b478cc887-trb6r"] Nov 03 18:13:54.053202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.053156 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:13:54.057611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.057550 4980 cpu_manager.go:414] "containerMap: removing container" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" Nov 03 18:13:54.057811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.057792 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" Nov 03 18:13:54.057942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.057930 4980 cpu_manager.go:414] "containerMap: removing container" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.058026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.058014 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.058139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.058102 4980 cpu_manager.go:414] "containerMap: removing container" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.058591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.058548 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.058972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.058954 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" containerName="payload-pre-processing" Nov 03 18:13:54.059093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.059082 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.059204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.059175 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="40a4fcb1-8f46-4bcd-8538-9c7734b42c73" containerName="istio-proxy" Nov 03 18:13:54.067696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.065839 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.093274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.093231 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp96x\" (UniqueName: \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.093458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.093437 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.108111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.108078 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:13:54.203692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.203642 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.203999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.203946 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp96x\" (UniqueName: \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.204848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.204819 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-43bdc0-dockercfg-mzd6k\"" Nov 03 18:13:54.204970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.204918 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-43bdc0\"" Nov 03 18:13:54.241598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.228658 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.303610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.301665 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp96x\" (UniqueName: \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\") pod \"payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.314109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.314068 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7bc4f39-5ff4-411d-85cf-a248ec146123" path="/var/lib/kubelet/pods/e7bc4f39-5ff4-411d-85cf-a248ec146123/volumes" Nov 03 18:13:54.314532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.314509 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:13:54.321904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.321876 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:13:54.322047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.321991 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.387239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.387193 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:54.414272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.407795 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.414272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.407928 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24trz\" (UniqueName: \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.509238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.509191 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.509611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.509588 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24trz\" (UniqueName: \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.510311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.510284 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.603734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.597004 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24trz\" (UniqueName: \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\") pod \"payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.644782 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.643135 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:54.991018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:54.990987 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:13:55.030636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:55.030604 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" event={"ID":"a4e58c41-9c10-43bc-8653-d1e616fc8060","Type":"ContainerStarted","Data":"870676bbdded96bc4453d734f38d784a6826a4f2cf159ee2aee0987046b31a28"} Nov 03 18:13:55.087327 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:55.087280 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:13:55.091700 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:55.091668 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b04e6e8_3c58_461d_adee_5d646fcfaa09.slice/crio-65ce778f8f2a823581c315e045cff17afa0f4bf362946bb504fab2424f3de4b2 WatchSource:0}: Error finding container 65ce778f8f2a823581c315e045cff17afa0f4bf362946bb504fab2424f3de4b2: Status 404 returned error can't find the container with id 65ce778f8f2a823581c315e045cff17afa0f4bf362946bb504fab2424f3de4b2 Nov 03 18:13:55.944533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:55.944497 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.119974 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120024 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120050 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120081 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120118 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120160 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120273 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120430 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120484 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.120513 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\") pod \"15237981-a676-4297-a2a2-4964991a41cf\" (UID: \"15237981-a676-4297-a2a2-4964991a41cf\") " Nov 03 18:13:56.127399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.121091 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.141698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.129281 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:56.141698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.129659 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:13:56.141698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.130730 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:56.141698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.130960 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:56.184810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.184754 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:13:56.190815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.190778 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "kube-api-access-kmqsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:56.199036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.198984 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:13:56.210103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.209969 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token" pod "15237981-a676-4297-a2a2-4964991a41cf" (UID: "15237981-a676-4297-a2a2-4964991a41cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:13:56.217472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.216483 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" event={"ID":"4b04e6e8-3c58-461d-adee-5d646fcfaa09","Type":"ContainerStarted","Data":"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f"} Nov 03 18:13:56.217472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.216543 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" event={"ID":"4b04e6e8-3c58-461d-adee-5d646fcfaa09","Type":"ContainerStarted","Data":"65ce778f8f2a823581c315e045cff17afa0f4bf362946bb504fab2424f3de4b2"} Nov 03 18:13:56.217472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.217417 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.221905 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.221946 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmqsb\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-kube-api-access-kmqsb\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.221963 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/15237981-a676-4297-a2a2-4964991a41cf-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.221980 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.221994 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222009 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/15237981-a676-4297-a2a2-4964991a41cf-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222025 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/15237981-a676-4297-a2a2-4964991a41cf-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222039 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/15237981-a676-4297-a2a2-4964991a41cf-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:13:56.222627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222541 4980 generic.go:356] "Generic (PLEG): container finished" podID="15237981-a676-4297-a2a2-4964991a41cf" containerID="04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7" exitCode=0 Nov 03 18:13:56.222683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222655 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerDied","Data":"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7"} Nov 03 18:13:56.222736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222698 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" event={"ID":"15237981-a676-4297-a2a2-4964991a41cf","Type":"ContainerDied","Data":"93c4a949923fd38396a183373b9995a3ae2b041b30f6cfdb9964997f3ee186bb"} Nov 03 18:13:56.222736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222731 4980 scope.go:122] "RemoveContainer" containerID="04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7" Nov 03 18:13:56.222868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.222856 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt" Nov 03 18:13:56.226469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.225844 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" event={"ID":"a4e58c41-9c10-43bc-8653-d1e616fc8060","Type":"ContainerStarted","Data":"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a"} Nov 03 18:13:56.226469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.226428 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:13:56.239142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.239121 4980 scope.go:122] "RemoveContainer" containerID="b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c" Nov 03 18:13:56.253469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.253452 4980 scope.go:122] "RemoveContainer" containerID="04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7" Nov 03 18:13:56.253773 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:56.253751 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7\": container with ID starting with 04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7 not found: ID does not exist" containerID="04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7" Nov 03 18:13:56.253857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.253779 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7"} err="failed to get container status \"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7\": rpc error: code = NotFound desc = could not find container \"04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7\": container with ID starting with 04eefcba24f6a123b51912f93a726a41671416efb844e9512820939a97ae33e7 not found: ID does not exist" Nov 03 18:13:56.253857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.253796 4980 scope.go:122] "RemoveContainer" containerID="b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c" Nov 03 18:13:56.254067 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:56.254047 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c\": container with ID starting with b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c not found: ID does not exist" containerID="b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c" Nov 03 18:13:56.254144 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.254069 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c"} err="failed to get container status \"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c\": rpc error: code = NotFound desc = could not find container \"b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c\": container with ID starting with b3b318f44e7ce7172d04bd49eb9c6896123d4c4308d411c033f6c5b358b6676c not found: ID does not exist" Nov 03 18:13:56.351320 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.351262 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" podStartSLOduration=2.351243328 podStartE2EDuration="2.351243328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:56.349607055 +0000 UTC m=+2416.178690654" watchObservedRunningTime="2026-11-03 18:13:56.351243328 +0000 UTC m=+2416.180326933" Nov 03 18:13:56.504614 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.499652 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:13:56.506167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.506112 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-b05459-openshift-default-7b4f876f6c-6hjgt"] Nov 03 18:13:56.606271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:56.606215 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" podStartSLOduration=3.606199014 podStartE2EDuration="3.606199014s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:13:56.605148397 +0000 UTC m=+2416.434232026" watchObservedRunningTime="2026-11-03 18:13:56.606199014 +0000 UTC m=+2416.435282616" Nov 03 18:13:57.746062 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.746029 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:13:57.746640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.746625 4980 cpu_manager.go:414] "containerMap: removing container" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:13:57.746699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.746641 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:13:57.746756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.746745 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:13:57.746799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.746760 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:13:57.751676 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.751650 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:57.883089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.883048 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-0bd8cf-serving-cert\"" Nov 03 18:13:57.890322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.890293 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:13:57.943134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.943091 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:57.943302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.943241 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:57.943302 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.943280 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:57.943548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:57.943525 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.045564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.044859 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.045810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.045788 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.045905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.045811 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.045971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.045910 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.045971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.045950 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.046523 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:58.046229 4980 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-serving-cert: secret "maas-api-e2e-api-a-0bd8cf-serving-cert" not found Nov 03 18:13:58.046523 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:13:58.046289 4980 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls podName:da64b884-5b99-4781-b5c0-e08695d422d6 nodeName:}" failed. No retries permitted until 2026-11-03 18:13:58.546271001 +0000 UTC m=+2418.375354605 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls") pod "maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" (UID: "da64b884-5b99-4781-b5c0-e08695d422d6") : secret "maas-api-e2e-api-a-0bd8cf-serving-cert" not found Nov 03 18:13:58.048617 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.048588 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.109538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.109284 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.302868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.302776 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="15237981-a676-4297-a2a2-4964991a41cf" path="/var/lib/kubelet/pods/15237981-a676-4297-a2a2-4964991a41cf/volumes" Nov 03 18:13:58.582057 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.581948 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.584818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.584787 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") pod \"maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:58.691249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:58.691214 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:13:59.084513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:59.084480 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:13:59.085776 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:13:59.085749 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda64b884_5b99_4781_b5c0_e08695d422d6.slice/crio-7b7783a321e662260454cc9e075586eb792da074620e03981e051b7274907675 WatchSource:0}: Error finding container 7b7783a321e662260454cc9e075586eb792da074620e03981e051b7274907675: Status 404 returned error can't find the container with id 7b7783a321e662260454cc9e075586eb792da074620e03981e051b7274907675 Nov 03 18:13:59.286649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:13:59.286600 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" event={"ID":"da64b884-5b99-4781-b5c0-e08695d422d6","Type":"ContainerStarted","Data":"7b7783a321e662260454cc9e075586eb792da074620e03981e051b7274907675"} Nov 03 18:14:00.296037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:00.295965 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" event={"ID":"da64b884-5b99-4781-b5c0-e08695d422d6","Type":"ContainerStarted","Data":"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c"} Nov 03 18:14:00.296037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:00.296030 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:14:00.418390 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:00.418344 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" podStartSLOduration=2.508088695 podStartE2EDuration="3.418322985s" totalImagesPullingTime="910.23429ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:13:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:00.416547235 +0000 UTC m=+2420.245630838" watchObservedRunningTime="2026-11-03 18:14:00.418322985 +0000 UTC m=+2420.247406587" Nov 03 18:14:02.161014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:02.160974 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:14:04.337155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.337063 4980 generic.go:356] "Generic (PLEG): container finished" podID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerID="eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888" exitCode=0 Nov 03 18:14:04.337660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.337146 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerDied","Data":"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888"} Nov 03 18:14:04.337737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.337696 4980 scope.go:122] "RemoveContainer" containerID="eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888" Nov 03 18:14:04.339225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.339200 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:04.339297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.339261 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:04.339297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:04.339293 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:05.345180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:05.345144 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerStarted","Data":"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f"} Nov 03 18:14:05.779229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:05.779193 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:14:05.782806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:05.782769 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:05.782994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:05.782843 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:06.321442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:06.321414 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:14:06.777131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:06.777092 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:06.777825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:06.777780 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:07.173661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.169656 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" podUID="8ef07506-cbf6-4349-b89b-d363751b1d0c" containerName="payload-pre-processing" containerID="cri-o://ad283f9fb720666197d26a75aeae710725fc151497b2a2279a31054cbd836444" gracePeriod=25 Nov 03 18:14:07.402162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.401730 4980 generic.go:356] "Generic (PLEG): container finished" podID="8ef07506-cbf6-4349-b89b-d363751b1d0c" containerID="ad283f9fb720666197d26a75aeae710725fc151497b2a2279a31054cbd836444" exitCode=0 Nov 03 18:14:07.402162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.401802 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" event={"ID":"8ef07506-cbf6-4349-b89b-d363751b1d0c","Type":"ContainerDied","Data":"ad283f9fb720666197d26a75aeae710725fc151497b2a2279a31054cbd836444"} Nov 03 18:14:07.508323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.508294 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:14:07.613734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.613690 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\" (UniqueName: \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\") pod \"8ef07506-cbf6-4349-b89b-d363751b1d0c\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " Nov 03 18:14:07.613944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.613924 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\") pod \"8ef07506-cbf6-4349-b89b-d363751b1d0c\" (UID: \"8ef07506-cbf6-4349-b89b-d363751b1d0c\") " Nov 03 18:14:07.614276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.614248 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume" pod "8ef07506-cbf6-4349-b89b-d363751b1d0c" (UID: "8ef07506-cbf6-4349-b89b-d363751b1d0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:07.616028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.616005 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c" pod "8ef07506-cbf6-4349-b89b-d363751b1d0c" (UID: "8ef07506-cbf6-4349-b89b-d363751b1d0c"). InnerVolumeSpecName "kube-api-access-hhw9c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:07.715389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.715267 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8ef07506-cbf6-4349-b89b-d363751b1d0c-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:07.715653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.715628 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhw9c\" (UniqueName: \"kubernetes.io/projected/8ef07506-cbf6-4349-b89b-d363751b1d0c-kube-api-access-hhw9c\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:07.777041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.776998 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:07.777368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:07.777338 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:08.283272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.283234 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:14:08.283638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.283606 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:14:08.411216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.411184 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" Nov 03 18:14:08.411395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.411246 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7" event={"ID":"8ef07506-cbf6-4349-b89b-d363751b1d0c","Type":"ContainerDied","Data":"a971f1459f75f48497a7f493e39b51142081f4335535c9d00d65e184e92d2633"} Nov 03 18:14:08.411395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.411291 4980 scope.go:122] "RemoveContainer" containerID="ad283f9fb720666197d26a75aeae710725fc151497b2a2279a31054cbd836444" Nov 03 18:14:08.689300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.689253 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:14:08.692041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.691998 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-640dc9-7c8dd744c9-2bwd7"] Nov 03 18:14:08.779821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.779766 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:08.780373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:08.779952 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:09.780037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:09.779994 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:09.780452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:09.780059 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:09.854079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:09.854048 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:14:10.296522 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:10.296108 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ef07506-cbf6-4349-b89b-d363751b1d0c" path="/var/lib/kubelet/pods/8ef07506-cbf6-4349-b89b-d363751b1d0c/volumes" Nov 03 18:14:10.774135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:10.774094 4980 patch_prober.go:29] interesting pod/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body= Nov 03 18:14:10.774352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:10.774150 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" Nov 03 18:14:11.779059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:11.779024 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:14:11.780158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:11.780136 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:14:14.524395 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:14:14.524347 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:14:16.434774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.434740 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:16.435490 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.435469 4980 cpu_manager.go:414] "containerMap: removing container" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:14:16.435550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.435493 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="15237981-a676-4297-a2a2-4964991a41cf" containerName="istio-proxy" Nov 03 18:14:16.435550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.435526 4980 cpu_manager.go:414] "containerMap: removing container" podUID="8ef07506-cbf6-4349-b89b-d363751b1d0c" containerName="payload-pre-processing" Nov 03 18:14:16.435550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.435532 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ef07506-cbf6-4349-b89b-d363751b1d0c" containerName="payload-pre-processing" Nov 03 18:14:16.435684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.435655 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ef07506-cbf6-4349-b89b-d363751b1d0c" containerName="payload-pre-processing" Nov 03 18:14:16.438342 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.438319 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.484864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.484832 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:16.487966 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.487946 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.499407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.499386 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-07f297-openshift-default-dockercfg-dfdjr\"" Nov 03 18:14:16.534766 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.534732 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:16.546868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.546842 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:16.587115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587081 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587124 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587204 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587240 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587334 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587363 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587396 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587422 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw8wn\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587501 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587526 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fj7r8\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587564 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587634 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587661 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587685 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587754 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587702 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587797 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.587864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587825 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.587918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.587896 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689200 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689169 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689215 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689265 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689304 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw8wn\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689374 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689482 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fj7r8\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689526 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689581 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689605 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689635 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689640 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689699 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689707 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.689829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689788 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689835 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689883 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689941 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.689981 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690023 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690032 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690079 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.690152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690094 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690165 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.690672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690495 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690672 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.690894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.690662 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.691192 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.691167 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.691519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.691495 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.692158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.692135 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.692683 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.692656 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.692860 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.692842 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.692909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.692862 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.752729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.752698 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.752853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.752747 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw8wn\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.752853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.752766 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:16.752935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.752884 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fj7r8\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\") pod \"e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:16.800592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:16.800534 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:17.050970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.050861 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:17.257207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.257175 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:17.263684 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:14:17.263653 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8318957_0c4a_48fa_8c17_8518ddd6c239.slice/crio-60e4686fe1dca68075bbed83286969e82dabd54c2018c4ce921aed5ab01d9cb6 WatchSource:0}: Error finding container 60e4686fe1dca68075bbed83286969e82dabd54c2018c4ce921aed5ab01d9cb6: Status 404 returned error can't find the container with id 60e4686fe1dca68075bbed83286969e82dabd54c2018c4ce921aed5ab01d9cb6 Nov 03 18:14:17.268370 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.268331 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.268455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.268418 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.268507 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.268457 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.445528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.445499 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:17.448809 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:14:17.448777 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb0ad984_dea1_4c69_a1f7_a25c908d221e.slice/crio-ba661aa0b0bdbbc5262a57bf0c6c420b306c050100ebd95ed850c1912dc85699 WatchSource:0}: Error finding container ba661aa0b0bdbbc5262a57bf0c6c420b306c050100ebd95ed850c1912dc85699: Status 404 returned error can't find the container with id ba661aa0b0bdbbc5262a57bf0c6c420b306c050100ebd95ed850c1912dc85699 Nov 03 18:14:17.451830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.451788 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.451958 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.451867 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.451958 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.451905 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:17.482823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.482756 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" event={"ID":"bb0ad984-dea1-4c69-a1f7-a25c908d221e","Type":"ContainerStarted","Data":"ba661aa0b0bdbbc5262a57bf0c6c420b306c050100ebd95ed850c1912dc85699"} Nov 03 18:14:17.485719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.485018 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" event={"ID":"a8318957-0c4a-48fa-8c17-8518ddd6c239","Type":"ContainerStarted","Data":"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848"} Nov 03 18:14:17.485719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.485066 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" event={"ID":"a8318957-0c4a-48fa-8c17-8518ddd6c239","Type":"ContainerStarted","Data":"60e4686fe1dca68075bbed83286969e82dabd54c2018c4ce921aed5ab01d9cb6"} Nov 03 18:14:17.611532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.610469 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podStartSLOduration=1.610449922 podStartE2EDuration="1.610449922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:17.61011206 +0000 UTC m=+2437.439195664" watchObservedRunningTime="2026-11-03 18:14:17.610449922 +0000 UTC m=+2437.439533529" Nov 03 18:14:17.800912 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:17.800875 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:18.516278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:18.516241 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" event={"ID":"bb0ad984-dea1-4c69-a1f7-a25c908d221e","Type":"ContainerStarted","Data":"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6"} Nov 03 18:14:18.699597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:18.699228 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podStartSLOduration=2.6992088340000002 podStartE2EDuration="2.699208834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:18.698142728 +0000 UTC m=+2438.527226360" watchObservedRunningTime="2026-11-03 18:14:18.699208834 +0000 UTC m=+2438.528292438" Nov 03 18:14:18.809654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:18.803338 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:18.809654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:18.803402 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:19.051459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:19.051424 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:19.052692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:19.052664 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:19.052760 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:19.052725 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:19.802043 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:19.801996 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:19.802700 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:19.802667 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:20.051844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:20.051813 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:20.052013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:20.051864 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:20.809597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:20.808956 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:20.809597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:20.809018 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:21.052235 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:21.052195 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:21.052428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:21.052255 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:21.816601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:21.802608 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:21.816601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:21.802669 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:22.052336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:22.052293 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:22.052550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:22.052361 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:22.801069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:22.801025 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:22.801376 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:22.801345 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:23.051689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:23.051596 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:23.051689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:23.051667 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:23.801237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:23.801199 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" start-of-body= Nov 03 18:14:23.801407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:23.801259 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.158:15021/healthz/ready\": dial tcp 10.132.0.158:15021: connect: connection refused" Nov 03 18:14:24.051880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:24.051781 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" start-of-body= Nov 03 18:14:24.051880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:24.051839 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.157:15021/healthz/ready\": dial tcp 10.132.0.157:15021: connect: connection refused" Nov 03 18:14:25.203638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.203601 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:25.204652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.204613 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:25.205626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.205601 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:25.233982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.233950 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:25.233982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.234809 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:25.238057 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:25.238034 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:26.443629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.443545 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:14:26.448497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.448449 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.503926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.503896 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-0adf17-openshift-default-dockercfg-tss2d\"" Nov 03 18:14:26.557730 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.557693 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:14:26.611366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.611291 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.611934 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.611834 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.612225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.612094 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613003 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613117 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613161 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613190 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613229 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jw4x\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.613504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.613268 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714281 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714193 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714339 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714454 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714496 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714526 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714617 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jw4x\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714660 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714760 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.714903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.714814 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.715130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.715106 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.715203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.715110 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.715203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.715142 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.715326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.715277 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.715443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.715423 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.717351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.717331 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.717441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.717396 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.777505 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.777477 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:26.777660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:26.777559 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jw4x\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\") pod \"e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:27.066844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.066749 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:27.520300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.520267 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:14:27.556174 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:14:27.556138 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c515d25_41c3_4e44_9a64_954be42e00cb.slice/crio-5eccd96a6502cf1490545234b6605f2c89ff35b59cd955cca5fb169495be3035 WatchSource:0}: Error finding container 5eccd96a6502cf1490545234b6605f2c89ff35b59cd955cca5fb169495be3035: Status 404 returned error can't find the container with id 5eccd96a6502cf1490545234b6605f2c89ff35b59cd955cca5fb169495be3035 Nov 03 18:14:27.559554 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.559517 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:27.559681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.559629 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:27.559723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.559680 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:14:27.593392 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:27.593355 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerStarted","Data":"5eccd96a6502cf1490545234b6605f2c89ff35b59cd955cca5fb169495be3035"} Nov 03 18:14:28.614191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:28.614150 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerStarted","Data":"368a4a0578f60464f70ce1b24e4a5fa774acc1b1d7a6f9f06613208f76d6ac77"} Nov 03 18:14:28.794703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:28.793619 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podStartSLOduration=2.793594414 podStartE2EDuration="2.793594414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:14:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:14:28.789256792 +0000 UTC m=+2448.618340397" watchObservedRunningTime="2026-11-03 18:14:28.793594414 +0000 UTC m=+2448.622678017" Nov 03 18:14:29.067619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:29.067556 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:29.073831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:29.073808 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:29.621418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:29.621386 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:29.622356 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:29.622336 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:14:39.125063 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:39.125021 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:39.126279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:39.126249 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" containerID="cri-o://4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848" gracePeriod=30 Nov 03 18:14:39.152667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:39.152632 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:39.152920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:39.152894 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" containerID="cri-o://bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6" gracePeriod=30 Nov 03 18:14:40.207184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:40.207148 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:14:40.207651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:40.207209 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:14:40.236913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:40.236880 4980 patch_prober.go:29] interesting pod/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:14:40.237049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:40.236933 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:14:44.446853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.446815 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:44.479305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.479283 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:44.520077 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:14:44.520040 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:14:44.594849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.594817 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.594849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.594855 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.594885 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.594958 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595001 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595022 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595050 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595072 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595131 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595153 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595179 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595229 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595302 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595334 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595403 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595414 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.595482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595452 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\") pod \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\" (UID: \"bb0ad984-dea1-4c69-a1f7-a25c908d221e\") " Nov 03 18:14:44.596156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595620 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.596156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595675 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\") pod \"a8318957-0c4a-48fa-8c17-8518ddd6c239\" (UID: \"a8318957-0c4a-48fa-8c17-8518ddd6c239\") " Nov 03 18:14:44.596156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.595947 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.596156 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596130 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.596371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596285 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:44.596434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596365 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.596434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596398 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.596434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596415 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.596744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596720 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.596947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596785 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:14:44.596947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.596930 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.597159 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.597132 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.597248 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.597139 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.597447 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.597404 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.598118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.598081 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:14:44.598266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.598241 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:44.599956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.599817 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:14:44.599956 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.599918 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "kube-api-access-fj7r8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:44.600137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.600114 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy" pod "bb0ad984-dea1-4c69-a1f7-a25c908d221e" (UID: "bb0ad984-dea1-4c69-a1f7-a25c908d221e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.600231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.600193 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "kube-api-access-lw8wn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:44.600231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.600216 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:14:44.600520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.600495 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy" pod "a8318957-0c4a-48fa-8c17-8518ddd6c239" (UID: "a8318957-0c4a-48fa-8c17-8518ddd6c239"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:14:44.698044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698011 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lw8wn\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-kube-api-access-lw8wn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698040 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698049 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698059 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fj7r8\" (UniqueName: \"kubernetes.io/projected/bb0ad984-dea1-4c69-a1f7-a25c908d221e-kube-api-access-fj7r8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698068 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698076 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698084 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698092 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698100 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698108 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698117 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a8318957-0c4a-48fa-8c17-8518ddd6c239-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698126 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698135 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698143 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a8318957-0c4a-48fa-8c17-8518ddd6c239-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.698267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.698151 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb0ad984-dea1-4c69-a1f7-a25c908d221e-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:14:44.728019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.727991 4980 generic.go:356] "Generic (PLEG): container finished" podID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerID="4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848" exitCode=0 Nov 03 18:14:44.728168 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.728051 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" Nov 03 18:14:44.728168 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.728070 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" event={"ID":"a8318957-0c4a-48fa-8c17-8518ddd6c239","Type":"ContainerDied","Data":"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848"} Nov 03 18:14:44.728168 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.728121 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22" event={"ID":"a8318957-0c4a-48fa-8c17-8518ddd6c239","Type":"ContainerDied","Data":"60e4686fe1dca68075bbed83286969e82dabd54c2018c4ce921aed5ab01d9cb6"} Nov 03 18:14:44.728168 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.728142 4980 scope.go:122] "RemoveContainer" containerID="4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848" Nov 03 18:14:44.729793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.729767 4980 generic.go:356] "Generic (PLEG): container finished" podID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerID="bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6" exitCode=0 Nov 03 18:14:44.729955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.729821 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" Nov 03 18:14:44.729955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.729841 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" event={"ID":"bb0ad984-dea1-4c69-a1f7-a25c908d221e","Type":"ContainerDied","Data":"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6"} Nov 03 18:14:44.729955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.729894 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr" event={"ID":"bb0ad984-dea1-4c69-a1f7-a25c908d221e","Type":"ContainerDied","Data":"ba661aa0b0bdbbc5262a57bf0c6c420b306c050100ebd95ed850c1912dc85699"} Nov 03 18:14:44.739615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.739565 4980 scope.go:122] "RemoveContainer" containerID="4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848" Nov 03 18:14:44.739970 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:14:44.739947 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848\": container with ID starting with 4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848 not found: ID does not exist" containerID="4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848" Nov 03 18:14:44.740038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.739979 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848"} err="failed to get container status \"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848\": rpc error: code = NotFound desc = could not find container \"4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848\": container with ID starting with 4cc94c7827e2182ac678d2aa793377b5014ba81249f257f9c3460afa3538f848 not found: ID does not exist" Nov 03 18:14:44.740038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.740000 4980 scope.go:122] "RemoveContainer" containerID="bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6" Nov 03 18:14:44.749541 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.749524 4980 scope.go:122] "RemoveContainer" containerID="bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6" Nov 03 18:14:44.750015 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:14:44.749946 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6\": container with ID starting with bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6 not found: ID does not exist" containerID="bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6" Nov 03 18:14:44.750015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.749984 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6"} err="failed to get container status \"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6\": rpc error: code = NotFound desc = could not find container \"bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6\": container with ID starting with bb377f34a2325e61b620bc97263c1d471bd785b6be02752237e724949560e7c6 not found: ID does not exist" Nov 03 18:14:44.868172 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.868135 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:44.877845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.877811 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vrtmr"] Nov 03 18:14:44.993806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.993713 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:44.998832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:44.998803 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-07f297-openshift-default-7b5575fcbc-vfv22"] Nov 03 18:14:46.295440 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:46.295404 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" path="/var/lib/kubelet/pods/a8318957-0c4a-48fa-8c17-8518ddd6c239/volumes" Nov 03 18:14:46.295936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:46.295920 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" path="/var/lib/kubelet/pods/bb0ad984-dea1-4c69-a1f7-a25c908d221e/volumes" Nov 03 18:14:55.538459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:14:55.538380 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 18:15:00.199311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.199274 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:00.199939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.199923 4980 cpu_manager.go:414] "containerMap: removing container" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" Nov 03 18:15:00.199997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.199941 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" Nov 03 18:15:00.199997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.199977 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" Nov 03 18:15:00.199997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.199983 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" Nov 03 18:15:00.200115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.200070 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8318957-0c4a-48fa-8c17-8518ddd6c239" containerName="istio-proxy" Nov 03 18:15:00.200115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.200082 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb0ad984-dea1-4c69-a1f7-a25c908d221e" containerName="istio-proxy" Nov 03 18:15:00.203820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.203796 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.257889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.257859 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:15:00.312553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.312517 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:00.330119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.330086 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.330389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.330369 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbs5g\" (UniqueName: \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.432013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.431984 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.432215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.432028 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbs5g\" (UniqueName: \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.432638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.432618 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.495332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.495268 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbs5g\" (UniqueName: \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\") pod \"maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.517399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.517373 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:00.895930 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:00.895898 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:00.897301 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:15:00.897273 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14957962_a62d_4262_b596_86aa06eae961.slice/crio-385b2ae761b620277e3574fdab90696ff9e234381eac90890b556210561366b2 WatchSource:0}: Error finding container 385b2ae761b620277e3574fdab90696ff9e234381eac90890b556210561366b2: Status 404 returned error can't find the container with id 385b2ae761b620277e3574fdab90696ff9e234381eac90890b556210561366b2 Nov 03 18:15:01.876518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:01.872808 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerStarted","Data":"f4e224c9b82fcd80c4ee04e3698f446aab6018582bdb59577793e1f25152f317"} Nov 03 18:15:01.876518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:01.872882 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerStarted","Data":"385b2ae761b620277e3574fdab90696ff9e234381eac90890b556210561366b2"} Nov 03 18:15:01.991368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:01.991329 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 18:15:02.038706 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:02.038402 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" podStartSLOduration=2.038383719 podStartE2EDuration="2.038383719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:02.037417142 +0000 UTC m=+2481.866500741" watchObservedRunningTime="2026-11-03 18:15:02.038383719 +0000 UTC m=+2481.867467322" Nov 03 18:15:02.043787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:02.043758 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 18:15:02.909624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:02.909034 4980 generic.go:356] "Generic (PLEG): container finished" podID="14957962-a62d-4262-b596-86aa06eae961" containerID="f4e224c9b82fcd80c4ee04e3698f446aab6018582bdb59577793e1f25152f317" exitCode=60 Nov 03 18:15:02.909624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:02.909211 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerDied","Data":"f4e224c9b82fcd80c4ee04e3698f446aab6018582bdb59577793e1f25152f317"} Nov 03 18:15:02.910092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:02.909659 4980 scope.go:122] "RemoveContainer" containerID="f4e224c9b82fcd80c4ee04e3698f446aab6018582bdb59577793e1f25152f317" Nov 03 18:15:03.926993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:03.926949 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerStarted","Data":"1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95"} Nov 03 18:15:04.831008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:04.830978 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 18:15:04.935205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:04.935170 4980 generic.go:356] "Generic (PLEG): container finished" podID="14957962-a62d-4262-b596-86aa06eae961" containerID="1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95" exitCode=60 Nov 03 18:15:04.935653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:04.935305 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerDied","Data":"1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95"} Nov 03 18:15:04.935653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:04.935347 4980 scope.go:122] "RemoveContainer" containerID="f4e224c9b82fcd80c4ee04e3698f446aab6018582bdb59577793e1f25152f317" Nov 03 18:15:04.935952 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:04.935932 4980 scope.go:122] "RemoveContainer" containerID="1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95" Nov 03 18:15:04.936536 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:04.936504 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk_odh-ai-gateway-infra(14957962-a62d-4262-b596-86aa06eae961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" podUID="14957962-a62d-4262-b596-86aa06eae961" Nov 03 18:15:05.944403 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:05.944370 4980 scope.go:122] "RemoveContainer" containerID="1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95" Nov 03 18:15:05.944839 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:05.944793 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk_odh-ai-gateway-infra(14957962-a62d-4262-b596-86aa06eae961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" podUID="14957962-a62d-4262-b596-86aa06eae961" Nov 03 18:15:06.931429 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:06.931377 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:15:06.935977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:06.935941 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:06.996245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:06.996216 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-44769b-openshift-default-dockercfg-clj8x\"" Nov 03 18:15:07.033059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.033012 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.040901 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.040858 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.041236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.041211 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.041383 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.041364 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.044600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.041508 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.044848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.044823 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.250902 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.250862 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f4h9\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.251291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.251268 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.251505 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.251484 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.251669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.106172 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:15:07.384681 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.355630 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.409517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.409470 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f4h9\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.412625 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.412739 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.412804 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.412983 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.413076 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.413104 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.413502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.413146 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.417986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.417944 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.420939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.418276 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.421075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.418683 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.421075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.418849 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.424595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.422854 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.433338 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.433302 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.439271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.439241 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.445525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.445488 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.490975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.490764 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f4h9\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\") pod \"e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:07.601214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:07.601131 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:08.083521 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:08.083484 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:15:08.085812 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:15:08.085778 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddeec6453_d8cd_4aae_b2f4_a8b91cb8702f.slice/crio-ebb2cdb4f3ae060a95444e5589d81afcf8b5bcea66999d4d91ef33513bdae6ff WatchSource:0}: Error finding container ebb2cdb4f3ae060a95444e5589d81afcf8b5bcea66999d4d91ef33513bdae6ff: Status 404 returned error can't find the container with id ebb2cdb4f3ae060a95444e5589d81afcf8b5bcea66999d4d91ef33513bdae6ff Nov 03 18:15:08.088823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:08.088789 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:08.088914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:08.088857 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:08.088914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:08.088886 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:09.001227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.001026 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerStarted","Data":"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64"} Nov 03 18:15:09.001227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.001095 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerStarted","Data":"ebb2cdb4f3ae060a95444e5589d81afcf8b5bcea66999d4d91ef33513bdae6ff"} Nov 03 18:15:09.031298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.031265 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:15:09.031780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.031747 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" containerID="cri-o://9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56" gracePeriod=30 Nov 03 18:15:09.051783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.051737 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:15:09.052046 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.052013 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" containerID="cri-o://60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d" gracePeriod=30 Nov 03 18:15:09.146808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.146745 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podStartSLOduration=3.146725126 podStartE2EDuration="3.146725126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:09.138262095 +0000 UTC m=+2488.967345698" watchObservedRunningTime="2026-11-03 18:15:09.146725126 +0000 UTC m=+2488.975808727" Nov 03 18:15:09.601941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.601897 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:09.608753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.608722 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:09.832173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:09.832117 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" containerID="cri-o://0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a" gracePeriod=25 Nov 03 18:15:10.009289 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.009251 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:10.010305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.010279 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:15:10.560553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.560508 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" containerID="cri-o://415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8" gracePeriod=45 Nov 03 18:15:10.939983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.939950 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:15:10.989217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989172 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.989426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989243 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.989426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989279 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.989426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989389 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\" (UniqueName: \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.989691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989515 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.989691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.989604 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\") pod \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\" (UID: \"cf38ea01-f628-4f52-91e4-58d8f2103bc4\") " Nov 03 18:15:10.991083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.990787 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:10.991721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.991686 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:10.991898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.991867 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:10.994196 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.994175 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:10.994991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.994962 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:10.995251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:10.995222 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257" pod "cf38ea01-f628-4f52-91e4-58d8f2103bc4" (UID: "cf38ea01-f628-4f52-91e4-58d8f2103bc4"). InnerVolumeSpecName "kube-api-access-cq257". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:11.020888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.020851 4980 generic.go:356] "Generic (PLEG): container finished" podID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerID="415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8" exitCode=0 Nov 03 18:15:11.022028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.022003 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" Nov 03 18:15:11.023004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.022980 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" event={"ID":"cf38ea01-f628-4f52-91e4-58d8f2103bc4","Type":"ContainerDied","Data":"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8"} Nov 03 18:15:11.023111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.023022 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz" event={"ID":"cf38ea01-f628-4f52-91e4-58d8f2103bc4","Type":"ContainerDied","Data":"dd6ccfa6c45fd7b4932eeeced9796ff32d6e7a595546f874702817d24cd43ce9"} Nov 03 18:15:11.023111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.023042 4980 scope.go:122] "RemoveContainer" containerID="415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8" Nov 03 18:15:11.040333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.040308 4980 scope.go:122] "RemoveContainer" containerID="415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8" Nov 03 18:15:11.040806 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:11.040779 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8\": container with ID starting with 415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8 not found: ID does not exist" containerID="415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8" Nov 03 18:15:11.040879 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.040819 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8"} err="failed to get container status \"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8\": rpc error: code = NotFound desc = could not find container \"415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8\": container with ID starting with 415f2ff544f914cc67587a535d48f7cbca3262c08cc216b2eb35867b8eeabcf8 not found: ID does not exist" Nov 03 18:15:11.091607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091537 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.091607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091588 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.091607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091602 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.091607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091612 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.092039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091624 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cf38ea01-f628-4f52-91e4-58d8f2103bc4-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.092039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.091637 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq257\" (UniqueName: \"kubernetes.io/projected/cf38ea01-f628-4f52-91e4-58d8f2103bc4-kube-api-access-cq257\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:11.188961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.188920 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 18:15:11.194044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:11.194004 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-548ca1/simulator-w5-548ca1-kserve-5894b98ddb-fxlsz"] Nov 03 18:15:12.296285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:12.296233 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" path="/var/lib/kubelet/pods/cf38ea01-f628-4f52-91e4-58d8f2103bc4/volumes" Nov 03 18:15:12.443750 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:12.443702 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:15:12.536158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:12.443775 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:15:12.536158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:12.524887 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:12.536158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:12.524949 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:13.576123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:13.576032 4980 patch_prober.go:29] interesting pod/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:15:13.576123 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:13.576106 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:15:14.045003 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.044972 4980 generic.go:356] "Generic (PLEG): container finished" podID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerID="fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07" exitCode=0 Nov 03 18:15:14.045205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.045040 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerDied","Data":"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07"} Nov 03 18:15:14.045507 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.045474 4980 scope.go:122] "RemoveContainer" containerID="fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07" Nov 03 18:15:14.047373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.047332 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.047489 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.047424 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.047489 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.047479 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.047830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.047807 4980 generic.go:356] "Generic (PLEG): container finished" podID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerID="d73a72c40c516cf032c174cc31cf35b175baa6344a13972c6618b307d16d29d2" exitCode=0 Nov 03 18:15:14.047932 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.047891 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerDied","Data":"d73a72c40c516cf032c174cc31cf35b175baa6344a13972c6618b307d16d29d2"} Nov 03 18:15:14.048245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.048229 4980 scope.go:122] "RemoveContainer" containerID="d73a72c40c516cf032c174cc31cf35b175baa6344a13972c6618b307d16d29d2" Nov 03 18:15:14.049511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.049482 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.049631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.049552 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.049696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.049635 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:14.446120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.445254 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:15:14.452831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.452803 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:15:14.529519 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:14.529484 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:15:14.566149 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566098 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.566149 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566150 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.566413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566211 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.566413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566255 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.566413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566291 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566546 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.567216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566623 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:14.567216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.566964 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.567216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567070 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567141 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567178 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567217 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567255 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567321 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567347 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567403 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567477 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567531 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.567913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567563 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567648 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\") pod \"91fd53ed-22db-4403-b2a8-ce80b2c393be\" (UID: \"91fd53ed-22db-4403-b2a8-ce80b2c393be\") " Nov 03 18:15:14.567913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567706 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\") pod \"a29da114-deaf-4a01-ae80-90cafa3b7c24\" (UID: \"a29da114-deaf-4a01-ae80-90cafa3b7c24\") " Nov 03 18:15:14.568151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.567959 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.568215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568167 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568386 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568463 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a29da114-deaf-4a01-ae80-90cafa3b7c24-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568487 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568503 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568518 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568535 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.568755 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.572548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.573243 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.574630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.573592 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:14.574630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.573814 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.575660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.575624 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.575968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.575931 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:14.576343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.576306 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:14.576960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.576843 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:14.576960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.576923 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv" pod "91fd53ed-22db-4403-b2a8-ce80b2c393be" (UID: "91fd53ed-22db-4403-b2a8-ce80b2c393be"). InnerVolumeSpecName "kube-api-access-jr5lv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:14.578971 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.578929 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:15:14.580724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.580703 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:14.580836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.580758 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4" pod "a29da114-deaf-4a01-ae80-90cafa3b7c24" (UID: "a29da114-deaf-4a01-ae80-90cafa3b7c24"). InnerVolumeSpecName "kube-api-access-s4ld4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670176 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670208 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670219 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670234 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670248 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670265 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/91fd53ed-22db-4403-b2a8-ce80b2c393be-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670278 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670298 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4ld4\" (UniqueName: \"kubernetes.io/projected/a29da114-deaf-4a01-ae80-90cafa3b7c24-kube-api-access-s4ld4\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670314 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670328 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jr5lv\" (UniqueName: \"kubernetes.io/projected/91fd53ed-22db-4403-b2a8-ce80b2c393be-kube-api-access-jr5lv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670351 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670372 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a29da114-deaf-4a01-ae80-90cafa3b7c24-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:14.670720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:14.670391 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/91fd53ed-22db-4403-b2a8-ce80b2c393be-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:15.109462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.108202 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerStarted","Data":"2dc9448b1cf4811a58c5429d6d63f73def0f3d38aa3d74b94207af07a9ad2f99"} Nov 03 18:15:15.129606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.128109 4980 generic.go:356] "Generic (PLEG): container finished" podID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerID="60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d" exitCode=0 Nov 03 18:15:15.129606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.128225 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" event={"ID":"a29da114-deaf-4a01-ae80-90cafa3b7c24","Type":"ContainerDied","Data":"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d"} Nov 03 18:15:15.129606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.128274 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" event={"ID":"a29da114-deaf-4a01-ae80-90cafa3b7c24","Type":"ContainerDied","Data":"9bd827750f626dad259d5c3dc74be4b683537f2d320d71b4d90837a92cba40c7"} Nov 03 18:15:15.129606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.128305 4980 scope.go:122] "RemoveContainer" containerID="60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d" Nov 03 18:15:15.129606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.128441 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f" Nov 03 18:15:15.176338 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.175359 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerStarted","Data":"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d"} Nov 03 18:15:15.351323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.334412 4980 scope.go:122] "RemoveContainer" containerID="60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d" Nov 03 18:15:15.370229 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:15.370127 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d\": container with ID starting with 60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d not found: ID does not exist" containerID="60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d" Nov 03 18:15:15.370477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.370446 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d"} err="failed to get container status \"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d\": rpc error: code = NotFound desc = could not find container \"60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d\": container with ID starting with 60009ba9de48de693f7f28ff675c503ede6b4ade3d799329308420935739ff8d not found: ID does not exist" Nov 03 18:15:15.403472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.403434 4980 generic.go:356] "Generic (PLEG): container finished" podID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerID="9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56" exitCode=0 Nov 03 18:15:15.408815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.408776 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerDied","Data":"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56"} Nov 03 18:15:15.409076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.409057 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" event={"ID":"91fd53ed-22db-4403-b2a8-ce80b2c393be","Type":"ContainerDied","Data":"276c6340cd3ada69310b4ba98e502858600548fcef7177ab7b75cc7d086358a9"} Nov 03 18:15:15.409193 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.409181 4980 scope.go:122] "RemoveContainer" containerID="9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56" Nov 03 18:15:15.409465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.409449 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97" Nov 03 18:15:15.425249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.425225 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:15:15.425356 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.425340 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:15.425418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.425384 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:15.434532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.434511 4980 scope.go:122] "RemoveContainer" containerID="6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297" Nov 03 18:15:15.445182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.445161 4980 scope.go:122] "RemoveContainer" containerID="9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56" Nov 03 18:15:15.445502 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:15.445479 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56\": container with ID starting with 9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56 not found: ID does not exist" containerID="9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56" Nov 03 18:15:15.445555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.445512 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56"} err="failed to get container status \"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56\": rpc error: code = NotFound desc = could not find container \"9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56\": container with ID starting with 9af053f4422daf1314c390969a223f2eacf2898c3c302d8f5609a5e8ef22ad56 not found: ID does not exist" Nov 03 18:15:15.445555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.445530 4980 scope.go:122] "RemoveContainer" containerID="6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297" Nov 03 18:15:15.445822 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:15.445797 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297\": container with ID starting with 6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297 not found: ID does not exist" containerID="6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297" Nov 03 18:15:15.445883 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.445833 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297"} err="failed to get container status \"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297\": rpc error: code = NotFound desc = could not find container \"6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297\": container with ID starting with 6bf5b84f2797f9b5bbe0f17414c5cb4e82c11dbff79b6fcc36615762e946e297 not found: ID does not exist" Nov 03 18:15:15.629328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.629233 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:15:15.635705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.635679 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-7rl9f"] Nov 03 18:15:15.906789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.906678 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:15:15.909442 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:15.909414 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6ed3e95c-openshift-default-7c74687597-s5n97"] Nov 03 18:15:16.007913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.007878 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:15:16.008974 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.008948 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 18:15:16.009117 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.008999 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 18:15:16.318494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.318458 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" path="/var/lib/kubelet/pods/91fd53ed-22db-4403-b2a8-ce80b2c393be/volumes" Nov 03 18:15:16.319178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.319155 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" path="/var/lib/kubelet/pods/a29da114-deaf-4a01-ae80-90cafa3b7c24/volumes" Nov 03 18:15:16.424552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.424522 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:16.424740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:16.424605 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:17.005987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.005947 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 18:15:17.006672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.006632 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 18:15:17.013526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.013496 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" containerID="cri-o://249dc588dafd433aa7bb84220b2e40470c473da460e17c829e554d0d8dde2888" gracePeriod=45 Nov 03 18:15:17.111225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.108220 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" containerID="cri-o://97c74595d86eed842748a7ad0311b4a73b1ad5cd3ef5b1e29bedb117233303f4" gracePeriod=45 Nov 03 18:15:17.230959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.230927 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused" Nov 03 18:15:17.231409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.231346 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" probeResult="failure" output="Get \"https://10.132.0.71:8000/ready\": dial tcp 10.132.0.71:8000: connect: connection refused" Nov 03 18:15:17.315657 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.314796 4980 scope.go:122] "RemoveContainer" containerID="1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95" Nov 03 18:15:17.482049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.471826 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:17.511664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.491044 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:17.740469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.740105 4980 generic.go:356] "Generic (PLEG): container finished" podID="1c3f6539-35b7-484e-ad83-c27292358b62" containerID="249dc588dafd433aa7bb84220b2e40470c473da460e17c829e554d0d8dde2888" exitCode=0 Nov 03 18:15:17.740469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.740397 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" event={"ID":"1c3f6539-35b7-484e-ad83-c27292358b62","Type":"ContainerDied","Data":"249dc588dafd433aa7bb84220b2e40470c473da460e17c829e554d0d8dde2888"} Nov 03 18:15:17.743504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.743475 4980 generic.go:356] "Generic (PLEG): container finished" podID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerID="97c74595d86eed842748a7ad0311b4a73b1ad5cd3ef5b1e29bedb117233303f4" exitCode=0 Nov 03 18:15:17.743993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.743969 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" event={"ID":"02a2ef7b-f802-4ae4-9967-c99c1340c3db","Type":"ContainerDied","Data":"97c74595d86eed842748a7ad0311b4a73b1ad5cd3ef5b1e29bedb117233303f4"} Nov 03 18:15:17.924992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:17.924936 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:15:18.040602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.023776 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:15:18.099813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.097193 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 18:15:18.099813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.097268 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 18:15:18.101335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.101290 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.101554 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.101533 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\" (UniqueName: \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.111735 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.108477 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.123760 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.123716 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.134246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.134209 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "kube-api-access-kfvgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:18.233274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.232369 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.233274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.232488 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.233274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.232658 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.233274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.232705 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\") pod \"1c3f6539-35b7-484e-ad83-c27292358b62\" (UID: \"1c3f6539-35b7-484e-ad83-c27292358b62\") " Nov 03 18:15:18.233274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.233187 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfvgh\" (UniqueName: \"kubernetes.io/projected/1c3f6539-35b7-484e-ad83-c27292358b62-kube-api-access-kfvgh\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.233878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.233369 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.233878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.233645 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.236504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.236184 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:15:18.238155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.238130 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.238155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.238135 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs" pod "1c3f6539-35b7-484e-ad83-c27292358b62" (UID: "1c3f6539-35b7-484e-ad83-c27292358b62"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:18.333903 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.333874 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\" (UniqueName: \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.334105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334089 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.334378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334354 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.334489 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334366 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\" (UniqueName: \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.334630 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334614 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.334797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334785 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.334928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.334915 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\") pod \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\" (UID: \"02a2ef7b-f802-4ae4-9967-c99c1340c3db\") " Nov 03 18:15:18.335449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.335422 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.335628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.335601 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.335895 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.335876 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.336075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.335978 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.336075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.336004 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.336075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.336022 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.336075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.336036 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1c3f6539-35b7-484e-ad83-c27292358b62-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.336075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.336050 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1c3f6539-35b7-484e-ad83-c27292358b62-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.337491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.337470 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:15:18.337686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.337665 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:15:18.337758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.337694 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc" pod "02a2ef7b-f802-4ae4-9967-c99c1340c3db" (UID: "02a2ef7b-f802-4ae4-9967-c99c1340c3db"). InnerVolumeSpecName "kube-api-access-n8nkc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:18.425295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.425204 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:18.425295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.425261 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:18.436988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.436959 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.436988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.436987 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8nkc\" (UniqueName: \"kubernetes.io/projected/02a2ef7b-f802-4ae4-9967-c99c1340c3db-kube-api-access-n8nkc\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.437219 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.436996 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/02a2ef7b-f802-4ae4-9967-c99c1340c3db-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.437219 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.437007 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/02a2ef7b-f802-4ae4-9967-c99c1340c3db-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:18.751864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.751842 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" Nov 03 18:15:18.752109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.751868 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k" event={"ID":"02a2ef7b-f802-4ae4-9967-c99c1340c3db","Type":"ContainerDied","Data":"aa26020d2796d4f7ad4aa4fa1700c91a00afafe11afdff037e246a433f726827"} Nov 03 18:15:18.752236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.752158 4980 scope.go:122] "RemoveContainer" containerID="97c74595d86eed842748a7ad0311b4a73b1ad5cd3ef5b1e29bedb117233303f4" Nov 03 18:15:18.753559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.753543 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" Nov 03 18:15:18.753848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.753595 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r" event={"ID":"1c3f6539-35b7-484e-ad83-c27292358b62","Type":"ContainerDied","Data":"55cdfb26909fa0c0de5dd54f974240a2337d6cfd9445b1061f73a2aaf198fbe7"} Nov 03 18:15:18.810939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.804658 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerStarted","Data":"c8c6265637996160e3ae66873f419ab57452fcce3f3c9d7199818d0584261f9c"} Nov 03 18:15:18.818232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.818185 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:15:18.827482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:18.824338 4980 scope.go:122] "RemoveContainer" containerID="249dc588dafd433aa7bb84220b2e40470c473da460e17c829e554d0d8dde2888" Nov 03 18:15:19.013663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.011736 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 18:15:19.013663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.011801 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 18:15:19.123596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.110821 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 18:15:19.130837 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.130807 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-w2-42c162-kserve-5c464d9f4f-brr6k"] Nov 03 18:15:19.236023 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.235990 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 18:15:19.246538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.246506 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-42c162/distinct-2-w2-42c162-kserve-9f47c85c4-r5t9r"] Nov 03 18:15:19.426953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.424325 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:19.426953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.425973 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:19.818038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.817996 4980 generic.go:356] "Generic (PLEG): container finished" podID="14957962-a62d-4262-b596-86aa06eae961" containerID="c8c6265637996160e3ae66873f419ab57452fcce3f3c9d7199818d0584261f9c" exitCode=60 Nov 03 18:15:19.818223 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.818073 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerDied","Data":"c8c6265637996160e3ae66873f419ab57452fcce3f3c9d7199818d0584261f9c"} Nov 03 18:15:19.818223 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.818158 4980 scope.go:122] "RemoveContainer" containerID="1db8db230954af09ef77320129e46cf59662dc3ce890f256844864d973ecfb95" Nov 03 18:15:19.818524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:19.818509 4980 scope.go:122] "RemoveContainer" containerID="c8c6265637996160e3ae66873f419ab57452fcce3f3c9d7199818d0584261f9c" Nov 03 18:15:19.818882 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:19.818860 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk_odh-ai-gateway-infra(14957962-a62d-4262-b596-86aa06eae961)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" podUID="14957962-a62d-4262-b596-86aa06eae961" Nov 03 18:15:20.012629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.009014 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" start-of-body= Nov 03 18:15:20.012629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.009071 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.153:15021/healthz/ready\": dial tcp 10.132.0.153:15021: connect: connection refused" Nov 03 18:15:20.014886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.014851 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:20.321662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.321617 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" path="/var/lib/kubelet/pods/02a2ef7b-f802-4ae4-9967-c99c1340c3db/volumes" Nov 03 18:15:20.322092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.322071 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" path="/var/lib/kubelet/pods/1c3f6539-35b7-484e-ad83-c27292358b62/volumes" Nov 03 18:15:20.424151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.424117 4980 patch_prober.go:29] interesting pod/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" start-of-body= Nov 03 18:15:20.424340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:20.424172 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.148:15021/healthz/ready\": dial tcp 10.132.0.148:15021: connect: connection refused" Nov 03 18:15:21.007383 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.007350 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:21.010236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.010215 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:15:21.011348 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.011331 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:15:21.114049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.113965 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\" (UniqueName: \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\") pod \"14957962-a62d-4262-b596-86aa06eae961\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " Nov 03 18:15:21.114213 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.114076 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\") pod \"14957962-a62d-4262-b596-86aa06eae961\" (UID: \"14957962-a62d-4262-b596-86aa06eae961\") " Nov 03 18:15:21.114469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.114440 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca" pod "14957962-a62d-4262-b596-86aa06eae961" (UID: "14957962-a62d-4262-b596-86aa06eae961"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:21.114645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.114533 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14957962-a62d-4262-b596-86aa06eae961-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.116253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.116231 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g" pod "14957962-a62d-4262-b596-86aa06eae961" (UID: "14957962-a62d-4262-b596-86aa06eae961"). InnerVolumeSpecName "kube-api-access-sbs5g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:21.216325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.216288 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbs5g\" (UniqueName: \"kubernetes.io/projected/14957962-a62d-4262-b596-86aa06eae961-kube-api-access-sbs5g\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:21.429594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.429508 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:15:21.429992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.429954 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:15:21.430826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.430805 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:15:21.836466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.836441 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" Nov 03 18:15:21.836699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.836478 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk" event={"ID":"14957962-a62d-4262-b596-86aa06eae961","Type":"ContainerDied","Data":"385b2ae761b620277e3574fdab90696ff9e234381eac90890b556210561366b2"} Nov 03 18:15:21.836699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:21.836543 4980 scope.go:122] "RemoveContainer" containerID="c8c6265637996160e3ae66873f419ab57452fcce3f3c9d7199818d0584261f9c" Nov 03 18:15:22.005077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:22.005040 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:22.015047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:22.015014 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-73a606-29895495-gl8gk"] Nov 03 18:15:22.296583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:22.296509 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14957962-a62d-4262-b596-86aa06eae961" path="/var/lib/kubelet/pods/14957962-a62d-4262-b596-86aa06eae961/volumes" Nov 03 18:15:28.793018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:28.792971 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:15:30.217419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.216950 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 18:15:30.287916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.287880 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\" (UniqueName: \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\") pod \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " Nov 03 18:15:30.288103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.288079 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\") pod \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\" (UID: \"3b09eb49-c2df-4df4-99cf-c5d1594dec9a\") " Nov 03 18:15:30.288404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.288382 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume" pod "3b09eb49-c2df-4df4-99cf-c5d1594dec9a" (UID: "3b09eb49-c2df-4df4-99cf-c5d1594dec9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:15:30.288589 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.288547 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:30.290761 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.290737 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684" pod "3b09eb49-c2df-4df4-99cf-c5d1594dec9a" (UID: "3b09eb49-c2df-4df4-99cf-c5d1594dec9a"). InnerVolumeSpecName "kube-api-access-wf684". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:15:30.390184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.390091 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf684\" (UniqueName: \"kubernetes.io/projected/3b09eb49-c2df-4df4-99cf-c5d1594dec9a-kube-api-access-wf684\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:15:30.900253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.900216 4980 generic.go:356] "Generic (PLEG): container finished" podID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerID="0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a" exitCode=0 Nov 03 18:15:30.900434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.900283 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" event={"ID":"3b09eb49-c2df-4df4-99cf-c5d1594dec9a","Type":"ContainerDied","Data":"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a"} Nov 03 18:15:30.900434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.900292 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" Nov 03 18:15:30.900434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.900339 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr" event={"ID":"3b09eb49-c2df-4df4-99cf-c5d1594dec9a","Type":"ContainerDied","Data":"67f8559490cbd2c54861aa4377069e0ce0e8c584e1788f49b139b7ec52815aff"} Nov 03 18:15:30.900434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.900369 4980 scope.go:122] "RemoveContainer" containerID="0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a" Nov 03 18:15:30.912273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.912247 4980 scope.go:122] "RemoveContainer" containerID="0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a" Nov 03 18:15:30.912618 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:30.912589 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a\": container with ID starting with 0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a not found: ID does not exist" containerID="0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a" Nov 03 18:15:30.912776 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:30.912747 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a"} err="failed to get container status \"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a\": rpc error: code = NotFound desc = could not find container \"0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a\": container with ID starting with 0ff7c6bdd2fba2a11d971e4de553e23ed4be063027199675d74e885c6b16e17a not found: ID does not exist" Nov 03 18:15:31.073782 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:31.073748 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 18:15:31.111535 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:31.111498 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-548ca1-7d597bb595-8f2vr"] Nov 03 18:15:32.296048 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:32.296003 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" path="/var/lib/kubelet/pods/3b09eb49-c2df-4df4-99cf-c5d1594dec9a/volumes" Nov 03 18:15:44.520362 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:15:44.520320 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:15:54.335863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.335832 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:15:54.337243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.337215 4980 cpu_manager.go:414] "containerMap: removing container" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.337411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.337396 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.337506 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.337495 4980 cpu_manager.go:414] "containerMap: removing container" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.341662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.341633 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.341829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.341813 4980 cpu_manager.go:414] "containerMap: removing container" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.341933 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.341919 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.342063 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342052 4980 cpu_manager.go:414] "containerMap: removing container" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" Nov 03 18:15:54.342151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342138 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" Nov 03 18:15:54.342291 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342274 4980 cpu_manager.go:414] "containerMap: removing container" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" Nov 03 18:15:54.342411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342395 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" Nov 03 18:15:54.342528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342514 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" Nov 03 18:15:54.342642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342628 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" Nov 03 18:15:54.342742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342731 4980 cpu_manager.go:414] "containerMap: removing container" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.342823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342813 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.342918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342908 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" Nov 03 18:15:54.342995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.342984 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" Nov 03 18:15:54.343088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343077 4980 cpu_manager.go:414] "containerMap: removing container" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.343170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343159 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.343260 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343249 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" Nov 03 18:15:54.343344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343333 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" Nov 03 18:15:54.343622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343605 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c3f6539-35b7-484e-ad83-c27292358b62" containerName="main" Nov 03 18:15:54.343746 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343734 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.343834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343824 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29da114-deaf-4a01-ae80-90cafa3b7c24" containerName="istio-proxy" Nov 03 18:15:54.343926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343916 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.344004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.343994 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="91fd53ed-22db-4403-b2a8-ce80b2c393be" containerName="istio-proxy" Nov 03 18:15:54.344090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.344080 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf38ea01-f628-4f52-91e4-58d8f2103bc4" containerName="main" Nov 03 18:15:54.344170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.344161 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a2ef7b-f802-4ae4-9967-c99c1340c3db" containerName="main" Nov 03 18:15:54.344248 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.344239 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b09eb49-c2df-4df4-99cf-c5d1594dec9a" containerName="payload-pre-processing" Nov 03 18:15:54.344325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.344315 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:15:54.354398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.354374 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.425794 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.425756 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:15:54.444710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.444671 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.450478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.450444 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452682 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452726 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452753 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btq7c\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452828 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452852 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452890 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.452964 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.457625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.449771 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ca3f31-openshift-default-dockercfg-q52h4\"" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561559 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561794 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561838 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561867 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561899 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btq7c\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.561974 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.562003 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.562047 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.562139 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.563939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.563231 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.567556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.567526 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.581504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.581472 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.588686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.581947 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.588944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.588534 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.589072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.588783 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.653195 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.653154 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.675472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.675406 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btq7c\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.684453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.684425 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\") pod \"e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:54.766443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:54.766156 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:55.339887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:55.339859 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:15:55.342940 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:15:55.342901 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25b0f430_91c2_4752_8e4e_7b065757e6f8.slice/crio-59490aef983e670a928cc8389ed0eb9609327b5acad25cc688a7823aa5e4e7e8 WatchSource:0}: Error finding container 59490aef983e670a928cc8389ed0eb9609327b5acad25cc688a7823aa5e4e7e8: Status 404 returned error can't find the container with id 59490aef983e670a928cc8389ed0eb9609327b5acad25cc688a7823aa5e4e7e8 Nov 03 18:15:55.346116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:55.346074 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:55.346237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:55.346166 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:55.346237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:55.346212 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:15:56.087280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:56.087238 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerStarted","Data":"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374"} Nov 03 18:15:56.087514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:56.087308 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerStarted","Data":"59490aef983e670a928cc8389ed0eb9609327b5acad25cc688a7823aa5e4e7e8"} Nov 03 18:15:56.226753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:56.226684 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podStartSLOduration=2.22666359 podStartE2EDuration="2.22666359s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:15:56.226630497 +0000 UTC m=+2536.055714100" watchObservedRunningTime="2026-11-03 18:15:56.22666359 +0000 UTC m=+2536.055747194" Nov 03 18:15:56.773610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:56.772069 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:56.791430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:56.791395 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:57.094134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:57.094052 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:15:57.095270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:15:57.095246 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:16:01.716078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.716037 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9"] Nov 03 18:16:01.716826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.716810 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="14957962-a62d-4262-b596-86aa06eae961" containerName="cleanup" Nov 03 18:16:01.719419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.719400 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.766198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.766167 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c"] Nov 03 18:16:01.768953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.768933 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:16:01.769097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.769033 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.771348 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.771327 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.789806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.789773 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.789975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.789828 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.789975 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.789895 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790016 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250bf139-b14a-4c76-8bf3-842f5be6494f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790081 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790160 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790186 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790225 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790221 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md64n\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-kube-api-access-md64n\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.790410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.790277 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.840598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.837501 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-zqhn9\"" Nov 03 18:16:01.868486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.866232 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9"] Nov 03 18:16:01.875864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.875774 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c"] Nov 03 18:16:01.878598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.878465 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:16:01.891271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891165 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6nc7\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.891271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891228 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.891271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891266 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891290 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891321 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891344 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891396 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250bf139-b14a-4c76-8bf3-842f5be6494f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891439 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891479 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.891615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891536 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891632 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891684 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891713 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891820 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp79f\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-kube-api-access-tp79f\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891868 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md64n\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-kube-api-access-md64n\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891909 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891946 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.891994 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892038 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892074 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892127 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892157 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892198 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892228 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892254 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892320 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892355 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819d1443-6698-423f-911d-8cb12ea7a3c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.892469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892409 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/250bf139-b14a-4c76-8bf3-842f5be6494f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.893047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.892879 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.893047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.893004 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.894470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.893505 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.894470 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.894184 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.895800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.895748 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.898043 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.898020 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.957827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.957794 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.958545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.958521 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-md64n\" (UniqueName: \"kubernetes.io/projected/250bf139-b14a-4c76-8bf3-842f5be6494f-kube-api-access-md64n\") pod \"maas-default-gateway-openshift-default-5df48fcc99-r96r9\" (UID: \"250bf139-b14a-4c76-8bf3-842f5be6494f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:01.994036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.993946 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.993989 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994013 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994032 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994058 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819d1443-6698-423f-911d-8cb12ea7a3c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994084 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6nc7\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994105 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994136 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994160 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994188 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994244 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994278 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994406 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994455 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp79f\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-kube-api-access-tp79f\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994466 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994511 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994536 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994548 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994660 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994704 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994861 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994864 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.994927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.994890 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/819d1443-6698-423f-911d-8cb12ea7a3c1-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.995643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.995048 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.995643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.995291 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.995762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.995718 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.996088 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.995982 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.996251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.996223 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.997326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.997296 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.997658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.997638 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:01.997800 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.997781 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:01.998215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:01.998196 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:02.037944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.037910 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:02.058629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.058589 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp79f\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-kube-api-access-tp79f\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:02.058813 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.058784 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:02.058995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.058974 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6nc7\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\") pod \"maas-default-gateway-openshift-default-5df48fcc99-q9c72\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:02.059065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.058973 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/819d1443-6698-423f-911d-8cb12ea7a3c1-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-4cj7c\" (UID: \"819d1443-6698-423f-911d-8cb12ea7a3c1\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:02.090810 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.090778 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:02.105138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.105113 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:02.490119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.490070 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9"] Nov 03 18:16:02.565832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.565787 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c"] Nov 03 18:16:02.584020 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:16:02.583768 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250bf139_b14a_4c76_8bf3_842f5be6494f.slice/crio-6d7173266cadf0932760f4fd42d15ace2ca46b6b7632cc294443636ad3315d11 WatchSource:0}: Error finding container 6d7173266cadf0932760f4fd42d15ace2ca46b6b7632cc294443636ad3315d11: Status 404 returned error can't find the container with id 6d7173266cadf0932760f4fd42d15ace2ca46b6b7632cc294443636ad3315d11 Nov 03 18:16:02.593045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.593007 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.593167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.593077 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.593167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.593110 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.593708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.593674 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.593944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.593905 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.594226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.594188 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.595901 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.595531 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:16:02.598553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.598518 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.598716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.598609 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:02.598716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:02.598652 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:03.182907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.182860 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" event={"ID":"a1c3f624-5986-47df-a72c-c0894ea403f9","Type":"ContainerStarted","Data":"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977"} Nov 03 18:16:03.183413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.182942 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" event={"ID":"a1c3f624-5986-47df-a72c-c0894ea403f9","Type":"ContainerStarted","Data":"221b9ebce63bdf9085cce7226700f264f2642115f5daedcd889d81ce98de1aaf"} Nov 03 18:16:03.186400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.186368 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" event={"ID":"250bf139-b14a-4c76-8bf3-842f5be6494f","Type":"ContainerStarted","Data":"72ffd4b20fdae11af2d14cdb901c25eecc2db4038f236606c6b127b456aa4a2b"} Nov 03 18:16:03.186531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.186435 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" event={"ID":"250bf139-b14a-4c76-8bf3-842f5be6494f","Type":"ContainerStarted","Data":"6d7173266cadf0932760f4fd42d15ace2ca46b6b7632cc294443636ad3315d11"} Nov 03 18:16:03.189135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.189110 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" event={"ID":"819d1443-6698-423f-911d-8cb12ea7a3c1","Type":"ContainerStarted","Data":"3582927cf1ef9b6649e1ef01799dff1ed2462adde66967e68d8bc687bf590bbf"} Nov 03 18:16:03.189261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.189151 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" event={"ID":"819d1443-6698-423f-911d-8cb12ea7a3c1","Type":"ContainerStarted","Data":"0dda8e2dbf29548dd06f697c5569bbd5c46bf9865b1b8698767ca1b9da291dcb"} Nov 03 18:16:03.311622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.308836 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podStartSLOduration=2.30881774 podStartE2EDuration="2.30881774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:03.308731859 +0000 UTC m=+2543.137815462" watchObservedRunningTime="2026-11-03 18:16:03.30881774 +0000 UTC m=+2543.137901345" Nov 03 18:16:03.445408 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.445342 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podStartSLOduration=2.445318758 podStartE2EDuration="2.445318758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:03.437738231 +0000 UTC m=+2543.266821832" watchObservedRunningTime="2026-11-03 18:16:03.445318758 +0000 UTC m=+2543.274402360" Nov 03 18:16:03.568778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:03.563299 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podStartSLOduration=2.56327836 podStartE2EDuration="2.56327836s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:16:03.561870375 +0000 UTC m=+2543.390953978" watchObservedRunningTime="2026-11-03 18:16:03.56327836 +0000 UTC m=+2543.392361957" Nov 03 18:16:04.041976 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.041941 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:04.043482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.043453 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:04.043671 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.043502 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:04.097017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.096970 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:04.104637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.099298 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:04.104637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.099348 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:04.131599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.109083 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:04.131599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.121664 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:04.131599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:04.121733 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:05.088599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.038290 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:05.088599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.038359 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:05.096599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.094947 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:05.096599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.095011 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:05.120598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.119722 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:05.120598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:05.119777 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:06.038508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.038471 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:06.038760 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.038529 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:06.095597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.093815 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:06.095597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.093879 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:06.107597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.106884 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:06.107597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:06.106938 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:07.039125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.039076 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:07.039334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.039136 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:07.100603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.100534 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:07.101263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.101219 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:07.106039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.105940 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:07.106039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:07.105983 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:08.038888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.038530 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:08.038888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.038611 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:08.140014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.133221 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:08.140014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.136682 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:08.140014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.140001 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:08.140677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:08.140061 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:09.038925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.038883 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:09.039112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.038941 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:09.092610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.092447 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:09.092610 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.092530 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:09.125599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.124959 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:09.125599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:09.125025 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:10.038876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.038834 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:10.039463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.038888 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:10.091480 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.091443 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" start-of-body= Nov 03 18:16:10.091692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.091509 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": dial tcp 10.132.0.164:15021: connect: connection refused" Nov 03 18:16:10.105824 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.105792 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:10.105990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.105852 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:10.402410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.402292 4980 generic.go:356] "Generic (PLEG): container finished" podID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerID="368a4a0578f60464f70ce1b24e4a5fa774acc1b1d7a6f9f06613208f76d6ac77" exitCode=0 Nov 03 18:16:10.402410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.402377 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerDied","Data":"368a4a0578f60464f70ce1b24e4a5fa774acc1b1d7a6f9f06613208f76d6ac77"} Nov 03 18:16:10.402830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.402813 4980 scope.go:122] "RemoveContainer" containerID="368a4a0578f60464f70ce1b24e4a5fa774acc1b1d7a6f9f06613208f76d6ac77" Nov 03 18:16:10.404702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.404665 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:10.404801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.404759 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:10.404842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:10.404808 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:11.038876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:11.038826 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" start-of-body= Nov 03 18:16:11.039449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:11.038908 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": dial tcp 10.132.0.163:15021: connect: connection refused" Nov 03 18:16:11.106228 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:11.106185 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-q9c72 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" start-of-body= Nov 03 18:16:11.106451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:11.106247 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.165:15021/healthz/ready\": dial tcp 10.132.0.165:15021: connect: connection refused" Nov 03 18:16:11.421867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:11.421554 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerStarted","Data":"996c50dcf9944f6b72a4a08d94b7d484266eb0b3734ed45f32db507d02024add"} Nov 03 18:16:12.075562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.075532 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:16:12.084736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.084700 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 18:16:12.085029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.084988 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 18:16:12.139986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.139657 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:16:12.139986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.139711 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:16:12.597599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.585920 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:12.597599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.587002 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:12.597599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.592843 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" Nov 03 18:16:12.622229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.622196 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:12.622948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.622924 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:12.623859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.623831 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" Nov 03 18:16:12.984425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.984393 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:12.986014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.985991 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:12.987253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:12.987234 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:16:13.086599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:13.075178 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 18:16:13.086599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:13.085690 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 18:16:14.075596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:14.074765 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 18:16:14.075596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:14.074844 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 18:16:14.520949 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:16:14.520913 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:16:14.621978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:14.621941 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:16:15.072004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:15.071964 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 18:16:15.072300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:15.072269 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 18:16:16.074189 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:16.074153 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" start-of-body= Nov 03 18:16:16.075274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:16.075241 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.159:15021/healthz/ready\": dial tcp 10.132.0.159:15021: connect: connection refused" Nov 03 18:16:17.073240 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:17.073204 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:16:17.074483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:17.074461 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:16:23.561378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:23.561342 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 18:16:23.610362 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:23.610320 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 18:16:26.051736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.051696 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:16:26.052139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.051753 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:16:26.580222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.580195 4980 generic.go:356] "Generic (PLEG): container finished" podID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerID="114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64" exitCode=0 Nov 03 18:16:26.580415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.580267 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerDied","Data":"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64"} Nov 03 18:16:26.580755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.580738 4980 scope.go:122] "RemoveContainer" containerID="114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64" Nov 03 18:16:26.582298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.582271 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:26.582371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.582341 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:26.582414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:26.582382 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:16:27.589587 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:27.589531 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerStarted","Data":"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893"} Nov 03 18:16:27.602009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:27.601987 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:16:27.602151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:27.602130 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:27.602207 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:27.602174 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:28.602109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:28.602068 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:28.602663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:28.602129 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:29.601696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:29.601664 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:29.601883 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:29.601714 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:30.587849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:30.587806 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 18:16:30.602202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:30.602166 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:30.602401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:30.602234 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:31.601645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:31.601592 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:31.602155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:31.601672 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:32.602430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:32.602387 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" start-of-body= Nov 03 18:16:32.602872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:32.602462 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.161:15021/healthz/ready\": dial tcp 10.132.0.161:15021: connect: connection refused" Nov 03 18:16:33.606138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:33.606108 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:16:33.606645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:33.606605 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:16:33.607318 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:33.607300 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:16:35.589089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:35.589048 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" containerID="cri-o://32cf9118c17ebf7e49fed046aaee339142bae0c7b7d030231c382dc618dbbfc6" gracePeriod=25 Nov 03 18:16:35.768280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:35.768236 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:16:38.585422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.585378 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" containerID="cri-o://eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da" gracePeriod=45 Nov 03 18:16:38.631076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.631016 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" containerID="cri-o://cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c" gracePeriod=45 Nov 03 18:16:38.830682 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.830644 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" probeResult="failure" output="Get \"https://10.132.0.59:8000/ready\": dial tcp 10.132.0.59:8000: connect: connection refused" Nov 03 18:16:38.936521 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.936493 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:16:38.940265 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.940243 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:16:38.999218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999186 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\" (UniqueName: \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:38.999396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999259 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:38.999396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999317 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:38.999396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999355 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\" (UniqueName: \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:38.999396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999385 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999414 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999464 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999540 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999564 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999635 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:38.999670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999657 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.000020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999680 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:39.000020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999886 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.000020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999833 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\") pod \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\" (UID: \"2a49f302-04b3-4ff0-b880-fde16f2c4b4a\") " Nov 03 18:16:39.000020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:38.999981 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.000020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000000 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\" (UniqueName: \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\") pod \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\" (UID: \"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59\") " Nov 03 18:16:39.000465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000087 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.000465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000096 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.000783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000701 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.000783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000779 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.000949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000794 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.000949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000807 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.000949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000821 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.000949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.000834 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.002701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.002661 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:16:39.002701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.002667 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:16:39.002882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.002697 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.002942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.002896 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:16:39.003051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.003034 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n" pod "2a49f302-04b3-4ff0-b880-fde16f2c4b4a" (UID: "2a49f302-04b3-4ff0-b880-fde16f2c4b4a"). InnerVolumeSpecName "kube-api-access-m6m8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:39.003139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.003125 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch" pod "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" (UID: "2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59"). InnerVolumeSpecName "kube-api-access-6fgch". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:39.102494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102458 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgch\" (UniqueName: \"kubernetes.io/projected/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-kube-api-access-6fgch\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.102494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102497 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6m8n\" (UniqueName: \"kubernetes.io/projected/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-kube-api-access-m6m8n\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.102736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102512 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.102736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102526 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.102736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102539 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.102736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.102552 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a49f302-04b3-4ff0-b880-fde16f2c4b4a-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:39.686298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.686265 4980 generic.go:356] "Generic (PLEG): container finished" podID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerID="eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da" exitCode=0 Nov 03 18:16:39.686834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.686316 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" event={"ID":"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59","Type":"ContainerDied","Data":"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da"} Nov 03 18:16:39.686834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.686339 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" Nov 03 18:16:39.686834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.686367 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk" event={"ID":"2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59","Type":"ContainerDied","Data":"53b1ead063da0016579bc9728fef23002c1c041add7360ea41e12976a853e002"} Nov 03 18:16:39.686834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.686388 4980 scope.go:122] "RemoveContainer" containerID="eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da" Nov 03 18:16:39.688199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.688177 4980 generic.go:356] "Generic (PLEG): container finished" podID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerID="cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c" exitCode=0 Nov 03 18:16:39.688296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.688222 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" Nov 03 18:16:39.688296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.688229 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" event={"ID":"2a49f302-04b3-4ff0-b880-fde16f2c4b4a","Type":"ContainerDied","Data":"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c"} Nov 03 18:16:39.688296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.688261 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8" event={"ID":"2a49f302-04b3-4ff0-b880-fde16f2c4b4a","Type":"ContainerDied","Data":"76bf32aa4d5e6229c45b01ea4bce09293b46b710f26919fd7e36bd0e2e1941e0"} Nov 03 18:16:39.698094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.698008 4980 scope.go:122] "RemoveContainer" containerID="eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da" Nov 03 18:16:39.698380 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:16:39.698348 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da\": container with ID starting with eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da not found: ID does not exist" containerID="eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da" Nov 03 18:16:39.698518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.698387 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da"} err="failed to get container status \"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da\": rpc error: code = NotFound desc = could not find container \"eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da\": container with ID starting with eac0e48155606b70a3ed6fb8fd6602cfe74b6f9a97f2bfb76d9f5882338ef8da not found: ID does not exist" Nov 03 18:16:39.698518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.698405 4980 scope.go:122] "RemoveContainer" containerID="cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c" Nov 03 18:16:39.707814 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.707791 4980 scope.go:122] "RemoveContainer" containerID="cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c" Nov 03 18:16:39.708118 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:16:39.708100 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c\": container with ID starting with cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c not found: ID does not exist" containerID="cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c" Nov 03 18:16:39.708167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.708124 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c"} err="failed to get container status \"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c\": rpc error: code = NotFound desc = could not find container \"cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c\": container with ID starting with cd326dde59a118f8b44e407807dae2bb5a85501096465645474684ff4009571c not found: ID does not exist" Nov 03 18:16:39.855632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.855540 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 18:16:39.862871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.862844 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/distinct-2-w1-c76280-kserve-659fff668f-j9wqk"] Nov 03 18:16:39.974058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.974023 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 18:16:39.981454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:39.981430 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-c76280/premium-w1-c76280-kserve-64ccf4ff5f-kbsk8"] Nov 03 18:16:40.295496 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:40.295461 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" path="/var/lib/kubelet/pods/2a49f302-04b3-4ff0-b880-fde16f2c4b4a/volumes" Nov 03 18:16:40.295951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:40.295936 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" path="/var/lib/kubelet/pods/2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59/volumes" Nov 03 18:16:43.638757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:43.638718 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:16:43.639224 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:43.638806 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:16:43.673594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:43.671721 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:16:43.673594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:43.671819 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:16:44.521101 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:16:44.521058 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:16:45.768867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:45.768825 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:16:55.922245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:55.922148 4980 generic.go:356] "Generic (PLEG): container finished" podID="648af06e-e212-42f7-a85b-62a350a6776f" containerID="32cf9118c17ebf7e49fed046aaee339142bae0c7b7d030231c382dc618dbbfc6" exitCode=0 Nov 03 18:16:55.922693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:55.922253 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" event={"ID":"648af06e-e212-42f7-a85b-62a350a6776f","Type":"ContainerDied","Data":"32cf9118c17ebf7e49fed046aaee339142bae0c7b7d030231c382dc618dbbfc6"} Nov 03 18:16:55.965606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:55.965558 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 18:16:56.065606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.065554 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\") pod \"648af06e-e212-42f7-a85b-62a350a6776f\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " Nov 03 18:16:56.065808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.065621 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\" (UniqueName: \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\") pod \"648af06e-e212-42f7-a85b-62a350a6776f\" (UID: \"648af06e-e212-42f7-a85b-62a350a6776f\") " Nov 03 18:16:56.066031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.066004 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume" pod "648af06e-e212-42f7-a85b-62a350a6776f" (UID: "648af06e-e212-42f7-a85b-62a350a6776f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:16:56.066191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.066171 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/648af06e-e212-42f7-a85b-62a350a6776f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:56.068065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.068039 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq" pod "648af06e-e212-42f7-a85b-62a350a6776f" (UID: "648af06e-e212-42f7-a85b-62a350a6776f"). InnerVolumeSpecName "kube-api-access-wsmdq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:16:56.167132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.167105 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wsmdq\" (UniqueName: \"kubernetes.io/projected/648af06e-e212-42f7-a85b-62a350a6776f-kube-api-access-wsmdq\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:16:56.770091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.770053 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.53:50052\" within 1s: context deadline exceeded" Nov 03 18:16:56.770277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.770141 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 18:16:56.770277 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:16:56.770167 4980 logging.go:55] [core] [Channel #3982 SubChannel #3983]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.53:50052", ServerName: "10.132.0.53:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.53:50052: operation was canceled" Nov 03 18:16:56.930021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.929995 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" Nov 03 18:16:56.930465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.930023 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5" event={"ID":"648af06e-e212-42f7-a85b-62a350a6776f","Type":"ContainerDied","Data":"4f5e5a9516672dd89cb7ee57a9f0aead32c2129ef3209b5b41e84ab77d13d0ac"} Nov 03 18:16:56.930465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:56.930075 4980 scope.go:122] "RemoveContainer" containerID="32cf9118c17ebf7e49fed046aaee339142bae0c7b7d030231c382dc618dbbfc6" Nov 03 18:16:57.098234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:57.098147 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 18:16:57.148420 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:57.148376 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-c76280-6847d8fd75-frrj5"] Nov 03 18:16:58.296798 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:16:58.296764 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648af06e-e212-42f7-a85b-62a350a6776f" path="/var/lib/kubelet/pods/648af06e-e212-42f7-a85b-62a350a6776f/volumes" Nov 03 18:17:02.487175 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:02.487146 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 18:17:02.487697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:02.487394 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" containerID="cri-o://13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83" gracePeriod=30 Nov 03 18:17:02.506822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:02.506791 4980 patch_prober.go:29] interesting pod/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:17:02.506932 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:02.506850 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:17:07.840049 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.840017 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 18:17:07.868184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868143 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868199 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868245 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868276 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868326 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868376 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868407 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868442 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868482 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\") pod \"468f327b-7814-4a45-b798-33ff5139b58e\" (UID: \"468f327b-7814-4a45-b798-33ff5139b58e\") " Nov 03 18:17:07.868871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868655 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:07.868871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.868796 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:07.869266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.869223 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:07.869266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.869246 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.869488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.869270 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.869488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.869347 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:07.869488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.869374 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:17:07.871051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.871001 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:17:07.871387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.871362 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:07.871530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.871501 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:17:07.871897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.871867 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd" pod "468f327b-7814-4a45-b798-33ff5139b58e" (UID: "468f327b-7814-4a45-b798-33ff5139b58e"). InnerVolumeSpecName "kube-api-access-6rlmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:07.970265 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970223 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970265 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970262 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rlmd\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-kube-api-access-6rlmd\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970277 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/468f327b-7814-4a45-b798-33ff5139b58e-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970291 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/468f327b-7814-4a45-b798-33ff5139b58e-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970304 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/468f327b-7814-4a45-b798-33ff5139b58e-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970316 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:07.970508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:07.970327 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/468f327b-7814-4a45-b798-33ff5139b58e-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:08.024426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.024388 4980 generic.go:356] "Generic (PLEG): container finished" podID="468f327b-7814-4a45-b798-33ff5139b58e" containerID="13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83" exitCode=0 Nov 03 18:17:08.028620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.024442 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" Nov 03 18:17:08.028620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.024484 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" event={"ID":"468f327b-7814-4a45-b798-33ff5139b58e","Type":"ContainerDied","Data":"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83"} Nov 03 18:17:08.028620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.024557 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555" event={"ID":"468f327b-7814-4a45-b798-33ff5139b58e","Type":"ContainerDied","Data":"1d0adf3519a043abb2cbb44c5dbb4ce3e8f66a22641c47077bf2b76a090abe98"} Nov 03 18:17:08.028620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.024625 4980 scope.go:122] "RemoveContainer" containerID="13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83" Nov 03 18:17:08.041284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.041260 4980 scope.go:122] "RemoveContainer" containerID="13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83" Nov 03 18:17:08.041672 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:17:08.041642 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83\": container with ID starting with 13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83 not found: ID does not exist" containerID="13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83" Nov 03 18:17:08.041786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.041685 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83"} err="failed to get container status \"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83\": rpc error: code = NotFound desc = could not find container \"13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83\": container with ID starting with 13f44ae3f501440bf01794722a7e5977fd1cb5a7a1efab55335c4d07e5537d83 not found: ID does not exist" Nov 03 18:17:08.195276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.195243 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 18:17:08.200743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.200718 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-42c162-openshift-default-845b7b7765-l6555"] Nov 03 18:17:08.295606 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:08.295555 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="468f327b-7814-4a45-b798-33ff5139b58e" path="/var/lib/kubelet/pods/468f327b-7814-4a45-b798-33ff5139b58e/volumes" Nov 03 18:17:14.521028 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:17:14.520987 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:17:23.821267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:23.821228 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:17:23.934796 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:23.934085 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:17:28.821672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:28.821623 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" podUID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" containerName="payload-processing" containerID="cri-o://3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d" gracePeriod=25 Nov 03 18:17:28.934922 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:28.934878 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" podUID="38533719-6e19-42e9-b8b7-cf6d00618c60" containerName="payload-pre-processing" containerID="cri-o://00cb819a60e7bd4a6b4af8404d187a74f2d70074118b3f2ff66bc67299736da8" gracePeriod=25 Nov 03 18:17:29.131308 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.131259 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:17:29.163169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.163129 4980 generic.go:356] "Generic (PLEG): container finished" podID="38533719-6e19-42e9-b8b7-cf6d00618c60" containerID="00cb819a60e7bd4a6b4af8404d187a74f2d70074118b3f2ff66bc67299736da8" exitCode=0 Nov 03 18:17:29.163358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.163211 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" event={"ID":"38533719-6e19-42e9-b8b7-cf6d00618c60","Type":"ContainerDied","Data":"00cb819a60e7bd4a6b4af8404d187a74f2d70074118b3f2ff66bc67299736da8"} Nov 03 18:17:29.168445 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.168411 4980 generic.go:356] "Generic (PLEG): container finished" podID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" containerID="3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d" exitCode=0 Nov 03 18:17:29.168627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.168597 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" event={"ID":"3bb4c043-bf27-488e-9007-6ca6cdc2034e","Type":"ContainerDied","Data":"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d"} Nov 03 18:17:29.168717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.168657 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" event={"ID":"3bb4c043-bf27-488e-9007-6ca6cdc2034e","Type":"ContainerDied","Data":"8d48b31f5bf608ed9d16f7d1ff2d3b2146940cdb1f68d7d7d3e8c62bf2f4c4e5"} Nov 03 18:17:29.168717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.168689 4980 scope.go:122] "RemoveContainer" containerID="3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d" Nov 03 18:17:29.168873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.168854 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66" Nov 03 18:17:29.188125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.187271 4980 scope.go:122] "RemoveContainer" containerID="3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d" Nov 03 18:17:29.188125 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:17:29.187701 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d\": container with ID starting with 3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d not found: ID does not exist" containerID="3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d" Nov 03 18:17:29.188125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.187749 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d"} err="failed to get container status \"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d\": rpc error: code = NotFound desc = could not find container \"3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d\": container with ID starting with 3f12971bf90cb30381b2e1d9fc86e9a9f1d865e8fc45a299cfe73c0f4e62484d not found: ID does not exist" Nov 03 18:17:29.212292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.212262 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\") pod \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " Nov 03 18:17:29.212476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.212344 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\" (UniqueName: \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\") pod \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\" (UID: \"3bb4c043-bf27-488e-9007-6ca6cdc2034e\") " Nov 03 18:17:29.213753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.213723 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume" pod "3bb4c043-bf27-488e-9007-6ca6cdc2034e" (UID: "3bb4c043-bf27-488e-9007-6ca6cdc2034e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:17:29.215025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.215002 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk" pod "3bb4c043-bf27-488e-9007-6ca6cdc2034e" (UID: "3bb4c043-bf27-488e-9007-6ca6cdc2034e"). InnerVolumeSpecName "kube-api-access-dhwlk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:29.237892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.237861 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:17:29.313098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.313066 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\") pod \"38533719-6e19-42e9-b8b7-cf6d00618c60\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " Nov 03 18:17:29.313272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.313161 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\" (UniqueName: \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\") pod \"38533719-6e19-42e9-b8b7-cf6d00618c60\" (UID: \"38533719-6e19-42e9-b8b7-cf6d00618c60\") " Nov 03 18:17:29.313536 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.313521 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3bb4c043-bf27-488e-9007-6ca6cdc2034e-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:29.313632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.313538 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhwlk\" (UniqueName: \"kubernetes.io/projected/3bb4c043-bf27-488e-9007-6ca6cdc2034e-kube-api-access-dhwlk\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:29.313632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.313536 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume" pod "38533719-6e19-42e9-b8b7-cf6d00618c60" (UID: "38533719-6e19-42e9-b8b7-cf6d00618c60"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:17:29.315532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.315507 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg" pod "38533719-6e19-42e9-b8b7-cf6d00618c60" (UID: "38533719-6e19-42e9-b8b7-cf6d00618c60"). InnerVolumeSpecName "kube-api-access-lf7mg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:17:29.414461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.414387 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38533719-6e19-42e9-b8b7-cf6d00618c60-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:29.414461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.414415 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf7mg\" (UniqueName: \"kubernetes.io/projected/38533719-6e19-42e9-b8b7-cf6d00618c60-kube-api-access-lf7mg\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:17:29.595959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.595923 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:17:29.599928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:29.599898 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-0bd8cf-7d84b94d69-ffb66"] Nov 03 18:17:30.177119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.177037 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" event={"ID":"38533719-6e19-42e9-b8b7-cf6d00618c60","Type":"ContainerDied","Data":"5d29b9fb917d5e7558f0ac7729c7964598b446bdb446f205175e4786ddd5f95d"} Nov 03 18:17:30.177119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.177063 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc" Nov 03 18:17:30.177119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.177097 4980 scope.go:122] "RemoveContainer" containerID="00cb819a60e7bd4a6b4af8404d187a74f2d70074118b3f2ff66bc67299736da8" Nov 03 18:17:30.294825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.294790 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" path="/var/lib/kubelet/pods/3bb4c043-bf27-488e-9007-6ca6cdc2034e/volumes" Nov 03 18:17:30.354714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.354672 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:17:30.359626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:30.359591 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-0bd8cf-6985fbfcbc-kkldc"] Nov 03 18:17:31.185098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.185071 4980 generic.go:356] "Generic (PLEG): container finished" podID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerID="b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374" exitCode=0 Nov 03 18:17:31.185665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.185150 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerDied","Data":"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374"} Nov 03 18:17:31.185746 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.185684 4980 scope.go:122] "RemoveContainer" containerID="b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374" Nov 03 18:17:31.187182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.187156 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:17:31.187259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.187226 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:17:31.187259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:31.187255 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:17:32.193623 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:32.193591 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerStarted","Data":"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b"} Nov 03 18:17:32.294150 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:32.294113 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38533719-6e19-42e9-b8b7-cf6d00618c60" path="/var/lib/kubelet/pods/38533719-6e19-42e9-b8b7-cf6d00618c60/volumes" Nov 03 18:17:32.770597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:32.770019 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:17:32.770597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:32.770187 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 18:17:32.770597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:32.770235 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 18:17:33.780602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:33.778072 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 18:17:33.780602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:33.778141 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 18:17:34.774598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:34.768084 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 18:17:34.774598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:34.768156 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 18:17:35.776599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:35.776070 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 18:17:35.776599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:35.776139 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 18:17:36.770080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:36.770038 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" start-of-body= Nov 03 18:17:36.770389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:36.770356 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.162:15021/healthz/ready\": dial tcp 10.132.0.162:15021: connect: connection refused" Nov 03 18:17:37.772108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:37.772071 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:17:37.772612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:37.772347 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:17:37.773113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:17:37.773094 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:17:44.519979 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:17:44.519934 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:18:00.611343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.611253 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 18:18:00.611861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.611495 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" podUID="ab978c3e-1202-4ffb-8281-cfb352c1741b" containerName="maas-api" containerID="cri-o://bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f" gracePeriod=30 Nov 03 18:18:00.912134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.912103 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 18:18:00.965475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.965440 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") pod \"ab978c3e-1202-4ffb-8281-cfb352c1741b\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " Nov 03 18:18:00.965690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.965652 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\") pod \"ab978c3e-1202-4ffb-8281-cfb352c1741b\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " Nov 03 18:18:00.965805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.965788 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\") pod \"ab978c3e-1202-4ffb-8281-cfb352c1741b\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " Nov 03 18:18:00.965878 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.965839 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\" (UniqueName: \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\") pod \"ab978c3e-1202-4ffb-8281-cfb352c1741b\" (UID: \"ab978c3e-1202-4ffb-8281-cfb352c1741b\") " Nov 03 18:18:00.966380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.966334 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca" pod "ab978c3e-1202-4ffb-8281-cfb352c1741b" (UID: "ab978c3e-1202-4ffb-8281-cfb352c1741b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:00.967999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.967973 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls" pod "ab978c3e-1202-4ffb-8281-cfb352c1741b" (UID: "ab978c3e-1202-4ffb-8281-cfb352c1741b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:18:00.968115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.968042 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs" pod "ab978c3e-1202-4ffb-8281-cfb352c1741b" (UID: "ab978c3e-1202-4ffb-8281-cfb352c1741b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:18:00.968322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:00.968299 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2" pod "ab978c3e-1202-4ffb-8281-cfb352c1741b" (UID: "ab978c3e-1202-4ffb-8281-cfb352c1741b"). InnerVolumeSpecName "kube-api-access-fknd2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:01.068033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.067993 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab978c3e-1202-4ffb-8281-cfb352c1741b-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.068033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.068028 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fknd2\" (UniqueName: \"kubernetes.io/projected/ab978c3e-1202-4ffb-8281-cfb352c1741b-kube-api-access-fknd2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.068312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.068054 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.068312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.068065 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ab978c3e-1202-4ffb-8281-cfb352c1741b-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:01.382329 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.382292 4980 generic.go:356] "Generic (PLEG): container finished" podID="ab978c3e-1202-4ffb-8281-cfb352c1741b" containerID="bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f" exitCode=0 Nov 03 18:18:01.382517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.382350 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" Nov 03 18:18:01.382517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.382374 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" event={"ID":"ab978c3e-1202-4ffb-8281-cfb352c1741b","Type":"ContainerDied","Data":"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f"} Nov 03 18:18:01.382517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.382431 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45" event={"ID":"ab978c3e-1202-4ffb-8281-cfb352c1741b","Type":"ContainerDied","Data":"7a98b43f396a6594f0e3208b2e14fd127e9a03d092bd6fbfc4b6fcf8e12ffb9b"} Nov 03 18:18:01.382517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.382454 4980 scope.go:122] "RemoveContainer" containerID="bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f" Nov 03 18:18:01.394262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.394244 4980 scope.go:122] "RemoveContainer" containerID="bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f" Nov 03 18:18:01.394554 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:01.394534 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f\": container with ID starting with bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f not found: ID does not exist" containerID="bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f" Nov 03 18:18:01.394627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.394580 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f"} err="failed to get container status \"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f\": rpc error: code = NotFound desc = could not find container \"bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f\": container with ID starting with bcc83cdafa2b8a91a6c792fc6763e42ce039f990cbc90adee2b592559932e39f not found: ID does not exist" Nov 03 18:18:01.553159 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.553118 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 18:18:01.629519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:01.629476 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w1-c76280-647df59b5d-d4m45"] Nov 03 18:18:02.294595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:02.294550 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab978c3e-1202-4ffb-8281-cfb352c1741b" path="/var/lib/kubelet/pods/ab978c3e-1202-4ffb-8281-cfb352c1741b/volumes" Nov 03 18:18:04.693277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:04.693232 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:18:04.807176 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:04.807137 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:18:09.694027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:09.693979 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" podUID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" containerName="payload-processing" containerID="cri-o://5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f" gracePeriod=25 Nov 03 18:18:09.807916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:09.807877 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" podUID="a4e58c41-9c10-43bc-8653-d1e616fc8060" containerName="payload-pre-processing" containerID="cri-o://6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a" gracePeriod=25 Nov 03 18:18:09.964596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:09.964491 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:18:10.085241 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.085193 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:18:10.119250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.119209 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\") pod \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " Nov 03 18:18:10.119460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.119320 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\" (UniqueName: \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\") pod \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\" (UID: \"4b04e6e8-3c58-461d-adee-5d646fcfaa09\") " Nov 03 18:18:10.119622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.119597 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume" pod "4b04e6e8-3c58-461d-adee-5d646fcfaa09" (UID: "4b04e6e8-3c58-461d-adee-5d646fcfaa09"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:10.119854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.119837 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b04e6e8-3c58-461d-adee-5d646fcfaa09-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:10.122073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.122045 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz" pod "4b04e6e8-3c58-461d-adee-5d646fcfaa09" (UID: "4b04e6e8-3c58-461d-adee-5d646fcfaa09"). InnerVolumeSpecName "kube-api-access-24trz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:10.220480 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.220394 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\") pod \"a4e58c41-9c10-43bc-8653-d1e616fc8060\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " Nov 03 18:18:10.220679 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.220536 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\" (UniqueName: \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\") pod \"a4e58c41-9c10-43bc-8653-d1e616fc8060\" (UID: \"a4e58c41-9c10-43bc-8653-d1e616fc8060\") " Nov 03 18:18:10.220896 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.220832 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume" pod "a4e58c41-9c10-43bc-8653-d1e616fc8060" (UID: "a4e58c41-9c10-43bc-8653-d1e616fc8060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:10.221103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.221081 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24trz\" (UniqueName: \"kubernetes.io/projected/4b04e6e8-3c58-461d-adee-5d646fcfaa09-kube-api-access-24trz\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:10.221197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.221110 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a4e58c41-9c10-43bc-8653-d1e616fc8060-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:10.223939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.223908 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x" pod "a4e58c41-9c10-43bc-8653-d1e616fc8060" (UID: "a4e58c41-9c10-43bc-8653-d1e616fc8060"). InnerVolumeSpecName "kube-api-access-xp96x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:10.322006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.321975 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp96x\" (UniqueName: \"kubernetes.io/projected/a4e58c41-9c10-43bc-8653-d1e616fc8060-kube-api-access-xp96x\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:10.465925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.465884 4980 generic.go:356] "Generic (PLEG): container finished" podID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" containerID="5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f" exitCode=0 Nov 03 18:18:10.466130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.465960 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" Nov 03 18:18:10.466130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.465960 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" event={"ID":"4b04e6e8-3c58-461d-adee-5d646fcfaa09","Type":"ContainerDied","Data":"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f"} Nov 03 18:18:10.466130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.466121 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt" event={"ID":"4b04e6e8-3c58-461d-adee-5d646fcfaa09","Type":"ContainerDied","Data":"65ce778f8f2a823581c315e045cff17afa0f4bf362946bb504fab2424f3de4b2"} Nov 03 18:18:10.466304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.466153 4980 scope.go:122] "RemoveContainer" containerID="5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f" Nov 03 18:18:10.467715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.467690 4980 generic.go:356] "Generic (PLEG): container finished" podID="a4e58c41-9c10-43bc-8653-d1e616fc8060" containerID="6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a" exitCode=0 Nov 03 18:18:10.467825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.467748 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" Nov 03 18:18:10.467825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.467759 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" event={"ID":"a4e58c41-9c10-43bc-8653-d1e616fc8060","Type":"ContainerDied","Data":"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a"} Nov 03 18:18:10.468091 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.468067 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55" event={"ID":"a4e58c41-9c10-43bc-8653-d1e616fc8060","Type":"ContainerDied","Data":"870676bbdded96bc4453d734f38d784a6826a4f2cf159ee2aee0987046b31a28"} Nov 03 18:18:10.476565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.476539 4980 scope.go:122] "RemoveContainer" containerID="5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f" Nov 03 18:18:10.476942 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:10.476915 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f\": container with ID starting with 5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f not found: ID does not exist" containerID="5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f" Nov 03 18:18:10.477061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.476960 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f"} err="failed to get container status \"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f\": rpc error: code = NotFound desc = could not find container \"5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f\": container with ID starting with 5d46796a6538722884a2e4589822f6ddb2747265c815e7a085f01845ccfeb73f not found: ID does not exist" Nov 03 18:18:10.477061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.476988 4980 scope.go:122] "RemoveContainer" containerID="6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a" Nov 03 18:18:10.488147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.488121 4980 scope.go:122] "RemoveContainer" containerID="6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a" Nov 03 18:18:10.488477 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:10.488458 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a\": container with ID starting with 6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a not found: ID does not exist" containerID="6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a" Nov 03 18:18:10.488546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.488490 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a"} err="failed to get container status \"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a\": rpc error: code = NotFound desc = could not find container \"6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a\": container with ID starting with 6e6a08924800f5cf51e536371cc8b48675395145a99663b9e406620f30a9ea9a not found: ID does not exist" Nov 03 18:18:10.638018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.637982 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:18:10.641016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.640986 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-43bdc0-595458b947-zbtbt"] Nov 03 18:18:10.763514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.763427 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:18:10.770984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:10.770947 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-43bdc0-74bbc768c-v2m55"] Nov 03 18:18:12.294872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:12.294830 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" path="/var/lib/kubelet/pods/4b04e6e8-3c58-461d-adee-5d646fcfaa09/volumes" Nov 03 18:18:12.295322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:12.295291 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4e58c41-9c10-43bc-8653-d1e616fc8060" path="/var/lib/kubelet/pods/a4e58c41-9c10-43bc-8653-d1e616fc8060/volumes" Nov 03 18:18:14.521162 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:14.521127 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:18:27.256203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.256170 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:18:27.256730 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.256408 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" podUID="da64b884-5b99-4781-b5c0-e08695d422d6" containerName="maas-api" containerID="cri-o://1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c" gracePeriod=30 Nov 03 18:18:27.550529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.550503 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:18:27.572459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.572432 4980 generic.go:356] "Generic (PLEG): container finished" podID="da64b884-5b99-4781-b5c0-e08695d422d6" containerID="1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c" exitCode=0 Nov 03 18:18:27.572632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.572482 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" Nov 03 18:18:27.572632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.572517 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" event={"ID":"da64b884-5b99-4781-b5c0-e08695d422d6","Type":"ContainerDied","Data":"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c"} Nov 03 18:18:27.572632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.572607 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p" event={"ID":"da64b884-5b99-4781-b5c0-e08695d422d6","Type":"ContainerDied","Data":"7b7783a321e662260454cc9e075586eb792da074620e03981e051b7274907675"} Nov 03 18:18:27.572632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.572632 4980 scope.go:122] "RemoveContainer" containerID="1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c" Nov 03 18:18:27.582205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.582188 4980 scope.go:122] "RemoveContainer" containerID="1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c" Nov 03 18:18:27.582499 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:27.582479 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c\": container with ID starting with 1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c not found: ID does not exist" containerID="1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c" Nov 03 18:18:27.582555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.582509 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c"} err="failed to get container status \"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c\": rpc error: code = NotFound desc = could not find container \"1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c\": container with ID starting with 1e369ff6c4ca1611a7694f1f2529b5b6970ca4547c0b9ef6b5757f2578d9025c not found: ID does not exist" Nov 03 18:18:27.697680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.697620 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\") pod \"da64b884-5b99-4781-b5c0-e08695d422d6\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " Nov 03 18:18:27.697680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.697671 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\") pod \"da64b884-5b99-4781-b5c0-e08695d422d6\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " Nov 03 18:18:27.697941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.697792 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\") pod \"da64b884-5b99-4781-b5c0-e08695d422d6\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " Nov 03 18:18:27.701013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.698648 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca" pod "da64b884-5b99-4781-b5c0-e08695d422d6" (UID: "da64b884-5b99-4781-b5c0-e08695d422d6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:27.701013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.698967 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") pod \"da64b884-5b99-4781-b5c0-e08695d422d6\" (UID: \"da64b884-5b99-4781-b5c0-e08695d422d6\") " Nov 03 18:18:27.701013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.699624 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da64b884-5b99-4781-b5c0-e08695d422d6-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:27.705523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.705479 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv" pod "da64b884-5b99-4781-b5c0-e08695d422d6" (UID: "da64b884-5b99-4781-b5c0-e08695d422d6"). InnerVolumeSpecName "kube-api-access-cnnjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:27.706267 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.706205 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs" pod "da64b884-5b99-4781-b5c0-e08695d422d6" (UID: "da64b884-5b99-4781-b5c0-e08695d422d6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:18:27.706941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.706911 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls" pod "da64b884-5b99-4781-b5c0-e08695d422d6" (UID: "da64b884-5b99-4781-b5c0-e08695d422d6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:18:27.804648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.804500 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnnjv\" (UniqueName: \"kubernetes.io/projected/da64b884-5b99-4781-b5c0-e08695d422d6-kube-api-access-cnnjv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:27.804648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.804546 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:27.804648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.804561 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da64b884-5b99-4781-b5c0-e08695d422d6-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:27.997163 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:27.997132 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:18:28.002386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:28.002359 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-0bd8cf-54bc7b599d-6r58p"] Nov 03 18:18:28.311973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:28.311842 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da64b884-5b99-4781-b5c0-e08695d422d6" path="/var/lib/kubelet/pods/da64b884-5b99-4781-b5c0-e08695d422d6/volumes" Nov 03 18:18:37.292919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:37.292880 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:18:37.293475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:37.293127 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" containerID="cri-o://14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d" gracePeriod=30 Nov 03 18:18:41.100353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:41.100328 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:18:41.105821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:41.105795 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:18:41.153698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:41.153674 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:18:41.158662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:41.158643 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:18:42.587153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.587127 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:18:42.666486 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.666437 4980 generic.go:356] "Generic (PLEG): container finished" podID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerID="14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d" exitCode=0 Nov 03 18:18:42.666680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.666513 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" Nov 03 18:18:42.666680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.666516 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerDied","Data":"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d"} Nov 03 18:18:42.666680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.666616 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t" event={"ID":"28e4954c-e9c7-406b-80b7-83b7d0e06d93","Type":"ContainerDied","Data":"ed4a6834cffe87e109563d2fc29ac32dc9d5e79a172118d741ee75476a6eb03f"} Nov 03 18:18:42.666680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.666645 4980 scope.go:122] "RemoveContainer" containerID="14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d" Nov 03 18:18:42.676926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.676906 4980 scope.go:122] "RemoveContainer" containerID="fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07" Nov 03 18:18:42.686639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.686614 4980 scope.go:122] "RemoveContainer" containerID="14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d" Nov 03 18:18:42.687004 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:42.686971 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d\": container with ID starting with 14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d not found: ID does not exist" containerID="14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d" Nov 03 18:18:42.687101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.687010 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d"} err="failed to get container status \"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d\": rpc error: code = NotFound desc = could not find container \"14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d\": container with ID starting with 14dd1893de9adab6ef968407f38ce944a836311b670fe3f75edf0d7717aadf3d not found: ID does not exist" Nov 03 18:18:42.687101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.687036 4980 scope.go:122] "RemoveContainer" containerID="fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07" Nov 03 18:18:42.687303 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:42.687282 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07\": container with ID starting with fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07 not found: ID does not exist" containerID="fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07" Nov 03 18:18:42.687402 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.687310 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07"} err="failed to get container status \"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07\": rpc error: code = NotFound desc = could not find container \"fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07\": container with ID starting with fe8c213e88f99bbae7b05dab33870e3253034fe60ff50ab9efb7a4257773fe07 not found: ID does not exist" Nov 03 18:18:42.714893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.714868 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.714944 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.714970 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715040 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715063 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715098 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715166 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715167 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715219 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715251 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\") pod \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\" (UID: \"28e4954c-e9c7-406b-80b7-83b7d0e06d93\") " Nov 03 18:18:42.715638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715378 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:42.715638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715469 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:18:42.715763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715717 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:42.716108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715895 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.716108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715925 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.716108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715943 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.716108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715957 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.716108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.715702 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:42.717421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.717399 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:42.717555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.717533 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:18:42.717628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.717553 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:18:42.717710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.717697 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm" pod "28e4954c-e9c7-406b-80b7-83b7d0e06d93" (UID: "28e4954c-e9c7-406b-80b7-83b7d0e06d93"). InnerVolumeSpecName "kube-api-access-2lprm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:18:42.817352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.817264 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.817352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.817298 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lprm\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-kube-api-access-2lprm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.817352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.817309 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.817352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.817318 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/28e4954c-e9c7-406b-80b7-83b7d0e06d93-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:42.817352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:42.817326 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/28e4954c-e9c7-406b-80b7-83b7d0e06d93-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:18:43.092190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:43.092093 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:18:43.097518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:43.097492 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-0bd8cf-openshift-default-bfbdc4455-j9l9t"] Nov 03 18:18:44.295008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:18:44.294977 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" path="/var/lib/kubelet/pods/28e4954c-e9c7-406b-80b7-83b7d0e06d93/volumes" Nov 03 18:18:44.519676 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:18:44.519634 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:19:14.520407 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:14.520369 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:19:19.352136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:19.352101 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 18:19:19.352665 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:19.352401 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" containerID="cri-o://12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b" gracePeriod=30 Nov 03 18:19:20.724846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:20.724772 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:19:20.725253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:20.724990 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" containerID="cri-o://2dc9448b1cf4811a58c5429d6d63f73def0f3d38aa3d74b94207af07a9ad2f99" gracePeriod=30 Nov 03 18:19:21.013246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:21.013156 4980 patch_prober.go:29] interesting pod/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:19:21.013246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:21.013223 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:19:24.624883 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.624828 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 18:19:24.637421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637391 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637543 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637496 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637617 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637561 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637618 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637680 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637704 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637702 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:24.637805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637781 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.637805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637793 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:24.638031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637814 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.638031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637873 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\") pod \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\" (UID: \"99ecb1af-07b7-474a-a3c2-a2eb50ddc736\") " Nov 03 18:19:24.638031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.637902 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:24.638197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638026 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:24.638197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638128 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:19:24.638484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638381 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.638484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638405 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.638484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638418 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.638484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638431 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.638484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.638445 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.640038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.640005 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:24.640203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.640185 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "kube-api-access-8hr8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:24.640331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.640314 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:24.640512 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.640497 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo" pod "99ecb1af-07b7-474a-a3c2-a2eb50ddc736" (UID: "99ecb1af-07b7-474a-a3c2-a2eb50ddc736"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:19:24.739842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.739816 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.739842 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.739840 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.740008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.739849 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.740008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.739858 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hr8v\" (UniqueName: \"kubernetes.io/projected/99ecb1af-07b7-474a-a3c2-a2eb50ddc736-kube-api-access-8hr8v\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:24.935593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.935482 4980 generic.go:356] "Generic (PLEG): container finished" podID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerID="12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b" exitCode=0 Nov 03 18:19:24.935593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.935539 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" Nov 03 18:19:24.935593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.935560 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerDied","Data":"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b"} Nov 03 18:19:24.935856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.935630 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn" event={"ID":"99ecb1af-07b7-474a-a3c2-a2eb50ddc736","Type":"ContainerDied","Data":"6c88c7473d9e75df04a314c18970ee6963e313058964978ba7e25faf26f9e9ee"} Nov 03 18:19:24.935856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.935654 4980 scope.go:122] "RemoveContainer" containerID="12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b" Nov 03 18:19:24.946583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.946543 4980 scope.go:122] "RemoveContainer" containerID="d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0" Nov 03 18:19:24.956593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.956553 4980 scope.go:122] "RemoveContainer" containerID="12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b" Nov 03 18:19:24.956851 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:24.956829 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b\": container with ID starting with 12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b not found: ID does not exist" containerID="12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b" Nov 03 18:19:24.956900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.956861 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b"} err="failed to get container status \"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b\": rpc error: code = NotFound desc = could not find container \"12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b\": container with ID starting with 12a5e712a2e59cb29f260ec0b084ce1c44355966fead602ad0940d23389bb49b not found: ID does not exist" Nov 03 18:19:24.956900 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.956880 4980 scope.go:122] "RemoveContainer" containerID="d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0" Nov 03 18:19:24.957112 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:24.957090 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0\": container with ID starting with d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0 not found: ID does not exist" containerID="d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0" Nov 03 18:19:24.957174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:24.957121 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0"} err="failed to get container status \"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0\": rpc error: code = NotFound desc = could not find container \"d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0\": container with ID starting with d0238188213f1033e26ef2b532dee39b73ee5bfdd7a73d3b9ed0531b611dd5a0 not found: ID does not exist" Nov 03 18:19:25.105827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:25.105796 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 18:19:25.114778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:25.114753 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-73a606-openshift-default-867f6586b7-mqjjn"] Nov 03 18:19:25.944266 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:25.944231 4980 generic.go:356] "Generic (PLEG): container finished" podID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerID="2dc9448b1cf4811a58c5429d6d63f73def0f3d38aa3d74b94207af07a9ad2f99" exitCode=0 Nov 03 18:19:25.944788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:25.944300 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerDied","Data":"2dc9448b1cf4811a58c5429d6d63f73def0f3d38aa3d74b94207af07a9ad2f99"} Nov 03 18:19:25.944788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:25.944373 4980 scope.go:122] "RemoveContainer" containerID="d73a72c40c516cf032c174cc31cf35b175baa6344a13972c6618b307d16d29d2" Nov 03 18:19:26.009506 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.009441 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:19:26.053717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053692 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.053837 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053807 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.053837 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053828 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.053937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053857 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.053999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053928 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:26.053999 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053969 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.054105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.053999 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.054105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054022 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.054105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054096 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.054262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054137 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\") pod \"759e0b50-5907-469c-b9b0-c6b1aca03260\" (UID: \"759e0b50-5907-469c-b9b0-c6b1aca03260\") " Nov 03 18:19:26.054262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054153 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:26.054386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054287 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:26.054386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054310 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:19:26.054495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054401 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:26.054950 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054923 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.055079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054952 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.055079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054980 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/759e0b50-5907-469c-b9b0-c6b1aca03260-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.055079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.054996 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.055079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.055010 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.056328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.056298 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:26.056428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.056346 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "kube-api-access-qp7hn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:26.056428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.056386 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:26.056428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.056393 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo" pod "759e0b50-5907-469c-b9b0-c6b1aca03260" (UID: "759e0b50-5907-469c-b9b0-c6b1aca03260"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:19:26.155689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.155660 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.155689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.155684 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qp7hn\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-kube-api-access-qp7hn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.155861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.155694 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.155861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.155706 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/759e0b50-5907-469c-b9b0-c6b1aca03260-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:26.300923 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.300849 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" path="/var/lib/kubelet/pods/99ecb1af-07b7-474a-a3c2-a2eb50ddc736/volumes" Nov 03 18:19:26.952745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.952706 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" Nov 03 18:19:26.952745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.952737 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq" event={"ID":"759e0b50-5907-469c-b9b0-c6b1aca03260","Type":"ContainerDied","Data":"49e0aba6f89be1af7c5cd2c8467f91b96447ab732682d81681e95c76bc32b166"} Nov 03 18:19:26.953295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:26.952801 4980 scope.go:122] "RemoveContainer" containerID="2dc9448b1cf4811a58c5429d6d63f73def0f3d38aa3d74b94207af07a9ad2f99" Nov 03 18:19:27.172237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:27.172200 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:19:27.223398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:27.223325 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-43bdc0-openshift-default-5bc7fc8c6f-gmfkq"] Nov 03 18:19:28.296391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:28.296352 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" path="/var/lib/kubelet/pods/759e0b50-5907-469c-b9b0-c6b1aca03260/volumes" Nov 03 18:19:31.322067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:31.322037 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 18:19:31.322495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:31.322264 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" containerID="cri-o://f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262" gracePeriod=30 Nov 03 18:19:36.596128 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.596069 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 18:19:36.683741 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683711 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.683905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683749 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.683905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683897 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683927 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683949 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.683974 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:36.684018 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684014 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684052 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684091 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684120 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\") pod \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\" (UID: \"1074adce-5d3a-4974-9ec1-c40f5abc4bbe\") " Nov 03 18:19:36.684387 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684298 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:36.684646 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684610 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:36.684755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684538 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:36.684755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684699 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.684755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.684723 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.685050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.685010 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:19:36.686348 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.686328 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:19:36.686613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.686555 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:19:36.686972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.686952 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "kube-api-access-lhdst". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:36.687032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.686974 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token" pod "1074adce-5d3a-4974-9ec1-c40f5abc4bbe" (UID: "1074adce-5d3a-4974-9ec1-c40f5abc4bbe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:19:36.785611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785546 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785612 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785628 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhdst\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-kube-api-access-lhdst\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785644 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785655 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785666 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:36.785833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:36.785678 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1074adce-5d3a-4974-9ec1-c40f5abc4bbe-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:19:37.038072 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.038037 4980 generic.go:356] "Generic (PLEG): container finished" podID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerID="f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262" exitCode=0 Nov 03 18:19:37.038253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.038084 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" Nov 03 18:19:37.038253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.038089 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerDied","Data":"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262"} Nov 03 18:19:37.038253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.038234 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds" event={"ID":"1074adce-5d3a-4974-9ec1-c40f5abc4bbe","Type":"ContainerDied","Data":"d99be589bd99610b88458257fe309c62aa5ba42009f88ae8def774612fe5d4b5"} Nov 03 18:19:37.038401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.038267 4980 scope.go:122] "RemoveContainer" containerID="f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262" Nov 03 18:19:37.049231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.049213 4980 scope.go:122] "RemoveContainer" containerID="ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd" Nov 03 18:19:37.059868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.059849 4980 scope.go:122] "RemoveContainer" containerID="f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262" Nov 03 18:19:37.060165 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:37.060147 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262\": container with ID starting with f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262 not found: ID does not exist" containerID="f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262" Nov 03 18:19:37.060221 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.060176 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262"} err="failed to get container status \"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262\": rpc error: code = NotFound desc = could not find container \"f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262\": container with ID starting with f9176af55aebbe4d3e7042d99194dd57e1f0f36a9fed5fe1e735bd4175cb9262 not found: ID does not exist" Nov 03 18:19:37.060221 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.060193 4980 scope.go:122] "RemoveContainer" containerID="ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd" Nov 03 18:19:37.060434 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:37.060419 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd\": container with ID starting with ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd not found: ID does not exist" containerID="ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd" Nov 03 18:19:37.060481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.060439 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd"} err="failed to get container status \"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd\": rpc error: code = NotFound desc = could not find container \"ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd\": container with ID starting with ac2262f28569babeb63bd14f3e897d36137d9a0627c2556927a21e2ce219cdbd not found: ID does not exist" Nov 03 18:19:37.207080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.207041 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 18:19:37.258503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:37.258469 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-962ds"] Nov 03 18:19:38.294178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:19:38.294140 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" path="/var/lib/kubelet/pods/1074adce-5d3a-4974-9ec1-c40f5abc4bbe/volumes" Nov 03 18:19:44.518729 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:19:44.518693 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:20:01.321345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:01.321310 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 18:20:01.321924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:01.321516 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" containerID="cri-o://2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e" gracePeriod=30 Nov 03 18:20:02.556935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:02.556905 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:20:02.557371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:02.557116 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" containerID="cri-o://55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893" gracePeriod=30 Nov 03 18:20:03.608967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:03.608926 4980 patch_prober.go:29] interesting pod/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:20:03.609373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:03.609007 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:20:06.589231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.589203 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 18:20:06.641704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641634 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.641861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641736 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.641861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641766 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.641861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641795 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.641861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641830 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.641861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641850 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.642160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641895 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.642160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641932 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.642160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.641968 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\") pod \"c574dffc-6350-48be-830c-d905dcbea1ea\" (UID: \"c574dffc-6350-48be-830c-d905dcbea1ea\") " Nov 03 18:20:06.642160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642086 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:06.642160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642095 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:06.642702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642386 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:06.642702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642403 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:06.642702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642425 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.642702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642444 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.642702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.642471 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:06.644263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.644236 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:06.644369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.644286 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:20:06.644369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.644306 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "kube-api-access-jjpdv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:06.644369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.644328 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token" pod "c574dffc-6350-48be-830c-d905dcbea1ea" (UID: "c574dffc-6350-48be-830c-d905dcbea1ea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:06.744021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.743991 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c574dffc-6350-48be-830c-d905dcbea1ea-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744019 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744028 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744039 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744049 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c574dffc-6350-48be-830c-d905dcbea1ea-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744058 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c574dffc-6350-48be-830c-d905dcbea1ea-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:06.744208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:06.744066 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjpdv\" (UniqueName: \"kubernetes.io/projected/c574dffc-6350-48be-830c-d905dcbea1ea-kube-api-access-jjpdv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.239529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.239497 4980 generic.go:356] "Generic (PLEG): container finished" podID="c574dffc-6350-48be-830c-d905dcbea1ea" containerID="2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e" exitCode=0 Nov 03 18:20:07.239714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.239543 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" Nov 03 18:20:07.239714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.239599 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" event={"ID":"c574dffc-6350-48be-830c-d905dcbea1ea","Type":"ContainerDied","Data":"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e"} Nov 03 18:20:07.239714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.239655 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl" event={"ID":"c574dffc-6350-48be-830c-d905dcbea1ea","Type":"ContainerDied","Data":"4e836a2b42cbc9e5b45d3ff157199dbfa784f28eba507c004465abb8875a6a1e"} Nov 03 18:20:07.239714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.239677 4980 scope.go:122] "RemoveContainer" containerID="2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e" Nov 03 18:20:07.251726 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.251700 4980 scope.go:122] "RemoveContainer" containerID="2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e" Nov 03 18:20:07.252056 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:07.252032 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e\": container with ID starting with 2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e not found: ID does not exist" containerID="2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e" Nov 03 18:20:07.252121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.252073 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e"} err="failed to get container status \"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e\": rpc error: code = NotFound desc = could not find container \"2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e\": container with ID starting with 2024bfdd1737ca6e773583c827a6940346688b216cd301327d03549154997a7e not found: ID does not exist" Nov 03 18:20:07.408343 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.408308 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 18:20:07.458396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.458361 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-776ba7-openshift-default-5f8667848c-vhgsl"] Nov 03 18:20:07.829255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.829194 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:20:07.856475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856450 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856485 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856504 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856625 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856682 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856744 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856799 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.856834 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856781 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:07.857070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856845 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.857070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856881 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\") pod \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\" (UID: \"deec6453-d8cd-4aae-b2f4-a8b91cb8702f\") " Nov 03 18:20:07.857070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856952 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:07.857070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.856982 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:07.857299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857196 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:07.857299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857245 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:07.857588 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857541 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.857717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857599 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.857717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857618 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.857717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857631 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.857717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.857645 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.858943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.858919 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "kube-api-access-4f4h9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:07.859040 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.858949 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:07.859093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.859079 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:20:07.859257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.859245 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy" pod "deec6453-d8cd-4aae-b2f4-a8b91cb8702f" (UID: "deec6453-d8cd-4aae-b2f4-a8b91cb8702f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:07.959093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.959066 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.959093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.959091 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.959280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.959101 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:07.959280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:07.959110 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4f4h9\" (UniqueName: \"kubernetes.io/projected/deec6453-d8cd-4aae-b2f4-a8b91cb8702f-kube-api-access-4f4h9\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:08.247220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.247187 4980 generic.go:356] "Generic (PLEG): container finished" podID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerID="55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893" exitCode=0 Nov 03 18:20:08.247422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.247241 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" Nov 03 18:20:08.247422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.247262 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerDied","Data":"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893"} Nov 03 18:20:08.247422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.247313 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2" event={"ID":"deec6453-d8cd-4aae-b2f4-a8b91cb8702f","Type":"ContainerDied","Data":"ebb2cdb4f3ae060a95444e5589d81afcf8b5bcea66999d4d91ef33513bdae6ff"} Nov 03 18:20:08.247422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.247336 4980 scope.go:122] "RemoveContainer" containerID="55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893" Nov 03 18:20:08.258415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.258396 4980 scope.go:122] "RemoveContainer" containerID="114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64" Nov 03 18:20:08.267753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.267727 4980 scope.go:122] "RemoveContainer" containerID="55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893" Nov 03 18:20:08.268033 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:08.268012 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893\": container with ID starting with 55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893 not found: ID does not exist" containerID="55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893" Nov 03 18:20:08.268073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.268044 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893"} err="failed to get container status \"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893\": rpc error: code = NotFound desc = could not find container \"55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893\": container with ID starting with 55e7edde11ad33f31bb6f9f9392008175b3c207647329365d6ffb6268439b893 not found: ID does not exist" Nov 03 18:20:08.268073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.268063 4980 scope.go:122] "RemoveContainer" containerID="114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64" Nov 03 18:20:08.268292 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:08.268277 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64\": container with ID starting with 114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64 not found: ID does not exist" containerID="114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64" Nov 03 18:20:08.268330 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.268299 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64"} err="failed to get container status \"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64\": rpc error: code = NotFound desc = could not find container \"114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64\": container with ID starting with 114395c14372fd6c3100637d271e355d26e82a5a87f373d09068288bf7133a64 not found: ID does not exist" Nov 03 18:20:08.293940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.293913 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" path="/var/lib/kubelet/pods/c574dffc-6350-48be-830c-d905dcbea1ea/volumes" Nov 03 18:20:08.417943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.417910 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:20:08.466174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:08.466140 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-44769b-openshift-default-58c7c9cc8c-wjsm2"] Nov 03 18:20:10.294669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:10.294637 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" path="/var/lib/kubelet/pods/deec6453-d8cd-4aae-b2f4-a8b91cb8702f/volumes" Nov 03 18:20:14.519432 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:14.519396 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:20:34.371561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:34.371521 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:20:34.372031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:34.371776 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" containerID="cri-o://de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b" gracePeriod=30 Nov 03 18:20:37.774518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:37.774480 4980 patch_prober.go:29] interesting pod/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:20:37.774959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:37.774538 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:20:39.644777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.644717 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:20:39.724414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724381 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724430 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724482 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724531 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724607 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724645 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724683 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.724788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724730 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.725004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724787 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\") pod \"25b0f430-91c2-4752-8e4e-7b065757e6f8\" (UID: \"25b0f430-91c2-4752-8e4e-7b065757e6f8\") " Nov 03 18:20:39.725004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724721 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:39.725004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724836 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:39.725004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.724893 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:39.725247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725048 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:39.725247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725152 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:39.725247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725230 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.725410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725261 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.725410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725276 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.725410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725285 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/25b0f430-91c2-4752-8e4e-7b065757e6f8-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.725410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.725294 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.727106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.727074 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:39.727236 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.727194 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:39.727406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.727389 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:20:39.727468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.727401 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c" pod "25b0f430-91c2-4752-8e4e-7b065757e6f8" (UID: "25b0f430-91c2-4752-8e4e-7b065757e6f8"). InnerVolumeSpecName "kube-api-access-btq7c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:39.826547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.826511 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.826547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.826546 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.826802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.826559 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.826802 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.826595 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btq7c\" (UniqueName: \"kubernetes.io/projected/25b0f430-91c2-4752-8e4e-7b065757e6f8-kube-api-access-btq7c\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:39.869898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:39.869863 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 18:20:40.451451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.451415 4980 generic.go:356] "Generic (PLEG): container finished" podID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerID="de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b" exitCode=0 Nov 03 18:20:40.451659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.451470 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" Nov 03 18:20:40.451659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.451491 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerDied","Data":"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b"} Nov 03 18:20:40.451659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.451545 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd" event={"ID":"25b0f430-91c2-4752-8e4e-7b065757e6f8","Type":"ContainerDied","Data":"59490aef983e670a928cc8389ed0eb9609327b5acad25cc688a7823aa5e4e7e8"} Nov 03 18:20:40.451659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.451581 4980 scope.go:122] "RemoveContainer" containerID="de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b" Nov 03 18:20:40.461447 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.461430 4980 scope.go:122] "RemoveContainer" containerID="b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374" Nov 03 18:20:40.470256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.470240 4980 scope.go:122] "RemoveContainer" containerID="de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b" Nov 03 18:20:40.470544 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:40.470525 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b\": container with ID starting with de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b not found: ID does not exist" containerID="de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b" Nov 03 18:20:40.470648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.470553 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b"} err="failed to get container status \"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b\": rpc error: code = NotFound desc = could not find container \"de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b\": container with ID starting with de6a82bdd7e3be2fa2e5e16642bd23d8b3763406ea1d487ad73c9e721095222b not found: ID does not exist" Nov 03 18:20:40.470648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.470585 4980 scope.go:122] "RemoveContainer" containerID="b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374" Nov 03 18:20:40.470858 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:40.470841 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374\": container with ID starting with b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374 not found: ID does not exist" containerID="b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374" Nov 03 18:20:40.470919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.470865 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374"} err="failed to get container status \"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374\": rpc error: code = NotFound desc = could not find container \"b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374\": container with ID starting with b27e4ee40dc3932c4ebc87fc2260bf55768835afa054468aa3f31e1a99078374 not found: ID does not exist" Nov 03 18:20:40.620273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.620193 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:20:40.670290 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:40.670253 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ca3f31-openshift-default-5dfbb6c89c-plqmd"] Nov 03 18:20:42.293367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:42.293330 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" path="/var/lib/kubelet/pods/25b0f430-91c2-4752-8e4e-7b065757e6f8/volumes" Nov 03 18:20:44.521377 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:44.521332 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:20:51.648179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:51.648096 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 18:20:51.648682 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:51.648328 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" podUID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" containerName="maas-api" containerID="cri-o://10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7" gracePeriod=30 Nov 03 18:20:51.958914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:51.958888 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 18:20:52.100732 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.100701 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") pod \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " Nov 03 18:20:52.100732 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.100739 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\" (UniqueName: \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\") pod \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " Nov 03 18:20:52.101015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.100870 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\") pod \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " Nov 03 18:20:52.101015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.100933 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\") pod \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\" (UID: \"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c\") " Nov 03 18:20:52.101386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.101355 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca" pod "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" (UID: "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:20:52.103118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.103095 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls" pod "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" (UID: "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:20:52.103583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.103549 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45" pod "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" (UID: "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c"). InnerVolumeSpecName "kube-api-access-mmk45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:52.103797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.103778 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs" pod "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" (UID: "022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:20:52.209347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.208699 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:52.209347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.208741 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmk45\" (UniqueName: \"kubernetes.io/projected/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-kube-api-access-mmk45\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:52.209347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.208757 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:52.209347 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.208771 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:52.529359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.529273 4980 generic.go:356] "Generic (PLEG): container finished" podID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" containerID="10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7" exitCode=0 Nov 03 18:20:52.529359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.529317 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" Nov 03 18:20:52.529613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.529351 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" event={"ID":"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c","Type":"ContainerDied","Data":"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7"} Nov 03 18:20:52.529613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.529406 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj" event={"ID":"022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c","Type":"ContainerDied","Data":"c21f923bcf50de147b48127e7873ce2fd2f20b81f072ad8ca55e61a069707899"} Nov 03 18:20:52.529613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.529427 4980 scope.go:122] "RemoveContainer" containerID="10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7" Nov 03 18:20:52.540856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.540835 4980 scope.go:122] "RemoveContainer" containerID="10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7" Nov 03 18:20:52.541182 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:52.541162 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7\": container with ID starting with 10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7 not found: ID does not exist" containerID="10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7" Nov 03 18:20:52.541244 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.541193 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7"} err="failed to get container status \"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7\": rpc error: code = NotFound desc = could not find container \"10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7\": container with ID starting with 10fa3c59d4334f2241100e180a66c067d2f9cb524aa158a96274b180910eb9e7 not found: ID does not exist" Nov 03 18:20:52.701182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.701142 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 18:20:52.706540 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:52.706512 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-f7d463-69cc8949c9-9wwjj"] Nov 03 18:20:54.292867 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:54.292832 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" path="/var/lib/kubelet/pods/022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c/volumes" Nov 03 18:20:54.891666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:54.891625 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" containerID="cri-o://ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11" gracePeriod=45 Nov 03 18:20:55.165967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.165940 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:20:55.255269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255236 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255271 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255298 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\" (UniqueName: \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255321 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255349 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\" (UniqueName: \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255400 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\") pod \"06c2c4c2-285a-4490-9a50-b653f4f808d5\" (UID: \"06c2c4c2-285a-4490-9a50-b653f4f808d5\") " Nov 03 18:20:55.255805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255610 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:55.255805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255636 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:55.255805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255616 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:55.255805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255781 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.255805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255793 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.256011 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.255808 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.257853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.257824 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "kube-api-access-x4q2b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:20:55.257984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.257829 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:20:55.257984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.257905 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm" pod "06c2c4c2-285a-4490-9a50-b653f4f808d5" (UID: "06c2c4c2-285a-4490-9a50-b653f4f808d5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:20:55.357641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.357610 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4q2b\" (UniqueName: \"kubernetes.io/projected/06c2c4c2-285a-4490-9a50-b653f4f808d5-kube-api-access-x4q2b\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.357641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.357639 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/06c2c4c2-285a-4490-9a50-b653f4f808d5-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.357641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.357650 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/06c2c4c2-285a-4490-9a50-b653f4f808d5-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:20:55.548543 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.548502 4980 generic.go:356] "Generic (PLEG): container finished" podID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerID="ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11" exitCode=0 Nov 03 18:20:55.548774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.548560 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" Nov 03 18:20:55.548774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.548602 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" event={"ID":"06c2c4c2-285a-4490-9a50-b653f4f808d5","Type":"ContainerDied","Data":"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11"} Nov 03 18:20:55.548774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.548655 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8" event={"ID":"06c2c4c2-285a-4490-9a50-b653f4f808d5","Type":"ContainerDied","Data":"36a042de23733a3fc346328012bb16ee882448e0366b23d4e98dabcfb4cbbcf5"} Nov 03 18:20:55.548774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.548680 4980 scope.go:122] "RemoveContainer" containerID="ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11" Nov 03 18:20:55.559533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.559508 4980 scope.go:122] "RemoveContainer" containerID="ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11" Nov 03 18:20:55.559871 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:20:55.559848 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11\": container with ID starting with ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11 not found: ID does not exist" containerID="ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11" Nov 03 18:20:55.559925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.559890 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11"} err="failed to get container status \"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11\": rpc error: code = NotFound desc = could not find container \"ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11\": container with ID starting with ed92f313266304825536d29e41c410b08f5b1ca2248d25da4519c9db5835fd11 not found: ID does not exist" Nov 03 18:20:55.716881 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.716836 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 18:20:55.766556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:55.766520 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-73a606/distinct-w4-73a606-kserve-7c4d5df658-mgdc8"] Nov 03 18:20:56.292514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:20:56.292483 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" path="/var/lib/kubelet/pods/06c2c4c2-285a-4490-9a50-b653f4f808d5/volumes" Nov 03 18:21:01.345525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.345484 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:01.346181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346161 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346185 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346203 4980 cpu_manager.go:414] "containerMap: removing container" podUID="ab978c3e-1202-4ffb-8281-cfb352c1741b" containerName="maas-api" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346210 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab978c3e-1202-4ffb-8281-cfb352c1741b" containerName="maas-api" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346228 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a4e58c41-9c10-43bc-8653-d1e616fc8060" containerName="payload-pre-processing" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346236 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4e58c41-9c10-43bc-8653-d1e616fc8060" containerName="payload-pre-processing" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346248 4980 cpu_manager.go:414] "containerMap: removing container" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346257 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346269 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:21:01.346280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346276 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346288 4980 cpu_manager.go:414] "containerMap: removing container" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346296 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346315 4980 cpu_manager.go:414] "containerMap: removing container" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346322 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346336 4980 cpu_manager.go:414] "containerMap: removing container" podUID="38533719-6e19-42e9-b8b7-cf6d00618c60" containerName="payload-pre-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346344 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38533719-6e19-42e9-b8b7-cf6d00618c60" containerName="payload-pre-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346369 4980 cpu_manager.go:414] "containerMap: removing container" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346376 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346387 4980 cpu_manager.go:414] "containerMap: removing container" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346393 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346403 4980 cpu_manager.go:414] "containerMap: removing container" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346409 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346422 4980 cpu_manager.go:414] "containerMap: removing container" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346429 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346437 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" containerName="payload-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346445 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" containerName="payload-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346466 4980 cpu_manager.go:414] "containerMap: removing container" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346473 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346485 4980 cpu_manager.go:414] "containerMap: removing container" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346493 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346509 4980 cpu_manager.go:414] "containerMap: removing container" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346516 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346527 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" containerName="payload-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346534 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" containerName="payload-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346543 4980 cpu_manager.go:414] "containerMap: removing container" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346550 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346583 4980 cpu_manager.go:414] "containerMap: removing container" podUID="da64b884-5b99-4781-b5c0-e08695d422d6" containerName="maas-api" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346591 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da64b884-5b99-4781-b5c0-e08695d422d6" containerName="maas-api" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346600 4980 cpu_manager.go:414] "containerMap: removing container" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346609 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346621 4980 cpu_manager.go:414] "containerMap: removing container" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346629 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346641 4980 cpu_manager.go:414] "containerMap: removing container" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346650 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346659 4980 cpu_manager.go:414] "containerMap: removing container" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346666 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346682 4980 cpu_manager.go:414] "containerMap: removing container" podUID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" containerName="maas-api" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346689 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" containerName="maas-api" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346699 4980 cpu_manager.go:414] "containerMap: removing container" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346706 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346833 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4e58c41-9c10-43bc-8653-d1e616fc8060" containerName="payload-pre-processing" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346848 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346858 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346866 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.346858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346877 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c2c4c2-285a-4490-9a50-b653f4f808d5" containerName="main" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346888 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="38533719-6e19-42e9-b8b7-cf6d00618c60" containerName="payload-pre-processing" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346900 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="99ecb1af-07b7-474a-a3c2-a2eb50ddc736" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346909 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a49f302-04b3-4ff0-b880-fde16f2c4b4a" containerName="main" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346919 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="468f327b-7814-4a45-b798-33ff5139b58e" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346928 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346937 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb4c043-bf27-488e-9007-6ca6cdc2034e" containerName="payload-processing" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346949 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="deec6453-d8cd-4aae-b2f4-a8b91cb8702f" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346958 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e4954c-e9c7-406b-80b7-83b7d0e06d93" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346969 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346979 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b04e6e8-3c58-461d-adee-5d646fcfaa09" containerName="payload-processing" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346988 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c574dffc-6350-48be-830c-d905dcbea1ea" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.346998 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="da64b884-5b99-4781-b5c0-e08695d422d6" containerName="maas-api" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347004 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="759e0b50-5907-469c-b9b0-c6b1aca03260" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347014 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="648af06e-e212-42f7-a85b-62a350a6776f" containerName="payload-pre-processing" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347025 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347035 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2eb99fa0-9a7e-4a6a-be3b-80043d5a3b59" containerName="main" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347044 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="022cffd6-d3ab-4eee-ba6a-f7d8be2f3c4c" containerName="maas-api" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347052 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="25b0f430-91c2-4752-8e4e-7b065757e6f8" containerName="istio-proxy" Nov 03 18:21:01.348983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.347060 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab978c3e-1202-4ffb-8281-cfb352c1741b" containerName="maas-api" Nov 03 18:21:01.352161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.352136 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.406923 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.406893 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-f7d463-openshift-default-dockercfg-wqnxq\"" Nov 03 18:21:01.421483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421447 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421486 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421508 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421650 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421703 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbdtr\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421814 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421801 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421819 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421840 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.421979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.421960 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.458719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.458681 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:01.523500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523461 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523506 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523534 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523643 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523695 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523725 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.523780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523758 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523802 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523859 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbdtr\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.523963 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.524131 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.524241 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524489 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.524466 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.524666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.524648 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.526668 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.526640 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.526960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.526938 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.586484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.586448 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.586648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.586605 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbdtr\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\") pod \"e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:01.667636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:01.667505 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:02.104059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.104037 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:02.106631 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:21:02.106603 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61cf8d9c_af3f_4c06_afbf_862c223c23d9.slice/crio-2517050a37100f19b2c659f664c5178b13359a80c6841111d8c73d6a811f1912 WatchSource:0}: Error finding container 2517050a37100f19b2c659f664c5178b13359a80c6841111d8c73d6a811f1912: Status 404 returned error can't find the container with id 2517050a37100f19b2c659f664c5178b13359a80c6841111d8c73d6a811f1912 Nov 03 18:21:02.109022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.108988 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:21:02.109125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.109069 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:21:02.109125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.109116 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:21:02.588065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.587988 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" event={"ID":"61cf8d9c-af3f-4c06-afbf-862c223c23d9","Type":"ContainerStarted","Data":"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178"} Nov 03 18:21:02.588065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.588050 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" event={"ID":"61cf8d9c-af3f-4c06-afbf-862c223c23d9","Type":"ContainerStarted","Data":"2517050a37100f19b2c659f664c5178b13359a80c6841111d8c73d6a811f1912"} Nov 03 18:21:02.667964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.667932 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:02.711811 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:02.711758 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" podStartSLOduration=1.711742555 podStartE2EDuration="1.711742555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:21:02.710661326 +0000 UTC m=+2842.539744969" watchObservedRunningTime="2026-11-03 18:21:02.711742555 +0000 UTC m=+2842.540826157" Nov 03 18:21:03.593263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:03.593218 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:03.600830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:03.600796 4980 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" secret="" err="secret \"e2e-worker-w6-f7d463-openshift-default-dockercfg-wqnxq\" not found" Nov 03 18:21:03.680321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:03.680278 4980 patch_prober.go:29] interesting pod/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" start-of-body= Nov 03 18:21:03.680701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:03.680668 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.166:15021/healthz/ready\": dial tcp 10.132.0.166:15021: connect: connection refused" Nov 03 18:21:04.605469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:04.605403 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerName="istio-proxy" containerID="cri-o://37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178" gracePeriod=30 Nov 03 18:21:14.422607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:14.422555 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:21:14.521476 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:14.521440 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:21:14.528765 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:14.528734 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:21:15.091317 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.091292 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:15.209785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.209752 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.209941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.209796 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.209941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.209858 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.209941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.209920 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.209946 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210009 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210013 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:15.210078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210034 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210074 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210163 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\") pod \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\" (UID: \"61cf8d9c-af3f-4c06-afbf-862c223c23d9\") " Nov 03 18:21:15.210355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210160 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:15.210355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210205 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:15.210355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210230 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:15.210355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210265 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:15.210626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210607 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.210696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210631 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.210696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210645 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.210696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210658 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.210696 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.210672 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.212368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.212337 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:21:15.212475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.212404 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:15.212933 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.212915 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "kube-api-access-nbdtr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:15.213010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.212934 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token" pod "61cf8d9c-af3f-4c06-afbf-862c223c23d9" (UID: "61cf8d9c-af3f-4c06-afbf-862c223c23d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:15.312189 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.312154 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.312189 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.312192 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.312386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.312207 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61cf8d9c-af3f-4c06-afbf-862c223c23d9-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.312386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.312221 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbdtr\" (UniqueName: \"kubernetes.io/projected/61cf8d9c-af3f-4c06-afbf-862c223c23d9-kube-api-access-nbdtr\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:15.666661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.666561 4980 generic.go:356] "Generic (PLEG): container finished" podID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerID="37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178" exitCode=0 Nov 03 18:21:15.666661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.666637 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" event={"ID":"61cf8d9c-af3f-4c06-afbf-862c223c23d9","Type":"ContainerDied","Data":"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178"} Nov 03 18:21:15.666661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.666638 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" Nov 03 18:21:15.667190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.666679 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr" event={"ID":"61cf8d9c-af3f-4c06-afbf-862c223c23d9","Type":"ContainerDied","Data":"2517050a37100f19b2c659f664c5178b13359a80c6841111d8c73d6a811f1912"} Nov 03 18:21:15.667190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.666699 4980 scope.go:122] "RemoveContainer" containerID="37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178" Nov 03 18:21:15.676755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.676738 4980 scope.go:122] "RemoveContainer" containerID="37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178" Nov 03 18:21:15.677030 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:15.677012 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178\": container with ID starting with 37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178 not found: ID does not exist" containerID="37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178" Nov 03 18:21:15.677075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.677042 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178"} err="failed to get container status \"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178\": rpc error: code = NotFound desc = could not find container \"37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178\": container with ID starting with 37c8e166bcf693cc588f2b19881d1cc2f3cd6ef2eec84709d364eb9dbe817178 not found: ID does not exist" Nov 03 18:21:15.837978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.837940 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:15.890752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:15.890684 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-f7d463-openshift-default-556cc75cbf-9jvlr"] Nov 03 18:21:16.293918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:16.293888 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" path="/var/lib/kubelet/pods/61cf8d9c-af3f-4c06-afbf-862c223c23d9/volumes" Nov 03 18:21:19.423415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.423342 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" podUID="6ecb4643-e2bf-4f7e-a110-f0927207496f" containerName="payload-processing" containerID="cri-o://300aeb2feace0c23f56ccd3897a705353233fd67d79e66f1316b50b101b011fb" gracePeriod=25 Nov 03 18:21:19.529562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.529523 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" containerID="cri-o://c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7" gracePeriod=25 Nov 03 18:21:19.692407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.692379 4980 generic.go:356] "Generic (PLEG): container finished" podID="6ecb4643-e2bf-4f7e-a110-f0927207496f" containerID="300aeb2feace0c23f56ccd3897a705353233fd67d79e66f1316b50b101b011fb" exitCode=0 Nov 03 18:21:19.692628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.692461 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" event={"ID":"6ecb4643-e2bf-4f7e-a110-f0927207496f","Type":"ContainerDied","Data":"300aeb2feace0c23f56ccd3897a705353233fd67d79e66f1316b50b101b011fb"} Nov 03 18:21:19.726859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.726833 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:21:19.783719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.783691 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\") pod \"6ecb4643-e2bf-4f7e-a110-f0927207496f\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " Nov 03 18:21:19.783844 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.783745 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\" (UniqueName: \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\") pod \"6ecb4643-e2bf-4f7e-a110-f0927207496f\" (UID: \"6ecb4643-e2bf-4f7e-a110-f0927207496f\") " Nov 03 18:21:19.784078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.784054 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume" pod "6ecb4643-e2bf-4f7e-a110-f0927207496f" (UID: "6ecb4643-e2bf-4f7e-a110-f0927207496f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:19.786161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.786137 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29" pod "6ecb4643-e2bf-4f7e-a110-f0927207496f" (UID: "6ecb4643-e2bf-4f7e-a110-f0927207496f"). InnerVolumeSpecName "kube-api-access-6cj29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:19.884723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.884684 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ecb4643-e2bf-4f7e-a110-f0927207496f-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:19.884723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:19.884718 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cj29\" (UniqueName: \"kubernetes.io/projected/6ecb4643-e2bf-4f7e-a110-f0927207496f-kube-api-access-6cj29\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:20.698996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:20.698968 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" Nov 03 18:21:20.698996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:20.698974 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6" event={"ID":"6ecb4643-e2bf-4f7e-a110-f0927207496f","Type":"ContainerDied","Data":"fd7a090d8f8ce1d0802766c9f54c529a32e534f3354c786f63732caf87688c0f"} Nov 03 18:21:20.699509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:20.699042 4980 scope.go:122] "RemoveContainer" containerID="300aeb2feace0c23f56ccd3897a705353233fd67d79e66f1316b50b101b011fb" Nov 03 18:21:20.869111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:20.869074 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:21:20.919773 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:20.919737 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-ca6b33-96c68cc5c-kprr6"] Nov 03 18:21:22.293817 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:22.293785 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ecb4643-e2bf-4f7e-a110-f0927207496f" path="/var/lib/kubelet/pods/6ecb4643-e2bf-4f7e-a110-f0927207496f/volumes" Nov 03 18:21:24.407313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.407271 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 18:21:24.407943 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.407535 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" podUID="4a5f7daa-bc77-49d6-af16-52075beb0338" containerName="maas-api" containerID="cri-o://539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1" gracePeriod=30 Nov 03 18:21:24.699525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.699499 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 18:21:24.722135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.722106 4980 generic.go:356] "Generic (PLEG): container finished" podID="4a5f7daa-bc77-49d6-af16-52075beb0338" containerID="539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1" exitCode=0 Nov 03 18:21:24.722295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.722150 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" Nov 03 18:21:24.722295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.722191 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" event={"ID":"4a5f7daa-bc77-49d6-af16-52075beb0338","Type":"ContainerDied","Data":"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1"} Nov 03 18:21:24.722295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.722258 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m" event={"ID":"4a5f7daa-bc77-49d6-af16-52075beb0338","Type":"ContainerDied","Data":"485275b328a7ccb00ef75d5d68811c539e44f841a337997153f7b9c7c8294446"} Nov 03 18:21:24.722295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.722288 4980 scope.go:122] "RemoveContainer" containerID="539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1" Nov 03 18:21:24.732246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.732229 4980 scope.go:122] "RemoveContainer" containerID="539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1" Nov 03 18:21:24.732517 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:24.732499 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1\": container with ID starting with 539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1 not found: ID does not exist" containerID="539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1" Nov 03 18:21:24.732605 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.732531 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1"} err="failed to get container status \"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1\": rpc error: code = NotFound desc = could not find container \"539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1\": container with ID starting with 539680af54cf47b97fe508697fd9971c53eb654e2231308fd0c75abe66c2d3c1 not found: ID does not exist" Nov 03 18:21:24.842106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842075 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\") pod \"4a5f7daa-bc77-49d6-af16-52075beb0338\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " Nov 03 18:21:24.842304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842140 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\") pod \"4a5f7daa-bc77-49d6-af16-52075beb0338\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " Nov 03 18:21:24.842304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842170 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") pod \"4a5f7daa-bc77-49d6-af16-52075beb0338\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " Nov 03 18:21:24.842304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842251 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\" (UniqueName: \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\") pod \"4a5f7daa-bc77-49d6-af16-52075beb0338\" (UID: \"4a5f7daa-bc77-49d6-af16-52075beb0338\") " Nov 03 18:21:24.842583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842534 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca" pod "4a5f7daa-bc77-49d6-af16-52075beb0338" (UID: "4a5f7daa-bc77-49d6-af16-52075beb0338"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:24.842777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.842756 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a5f7daa-bc77-49d6-af16-52075beb0338-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:24.844441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.844417 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs" pod "4a5f7daa-bc77-49d6-af16-52075beb0338" (UID: "4a5f7daa-bc77-49d6-af16-52075beb0338"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:21:24.844987 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.844965 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh" pod "4a5f7daa-bc77-49d6-af16-52075beb0338" (UID: "4a5f7daa-bc77-49d6-af16-52075beb0338"). InnerVolumeSpecName "kube-api-access-7l8lh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:24.845122 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.845101 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls" pod "4a5f7daa-bc77-49d6-af16-52075beb0338" (UID: "4a5f7daa-bc77-49d6-af16-52075beb0338"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:21:24.949135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.949027 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:24.949135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.949069 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4a5f7daa-bc77-49d6-af16-52075beb0338-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:24.949135 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:24.949087 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7l8lh\" (UniqueName: \"kubernetes.io/projected/4a5f7daa-bc77-49d6-af16-52075beb0338-kube-api-access-7l8lh\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:25.146871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:25.146833 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 18:21:25.152957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:25.152932 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-73a606-8f4cb56c4-t8j4m"] Nov 03 18:21:26.293440 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:26.293407 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5f7daa-bc77-49d6-af16-52075beb0338" path="/var/lib/kubelet/pods/4a5f7daa-bc77-49d6-af16-52075beb0338/volumes" Nov 03 18:21:28.927565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:28.927531 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:21:31.075642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.075606 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:21:31.076111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.075807 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" podUID="4ad45e40-7c86-4b9e-919c-8b742871defc" containerName="maas-api" containerID="cri-o://7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef" gracePeriod=30 Nov 03 18:21:31.367136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.367111 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:21:31.517982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.517942 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\") pod \"4ad45e40-7c86-4b9e-919c-8b742871defc\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " Nov 03 18:21:31.518164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.518047 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") pod \"4ad45e40-7c86-4b9e-919c-8b742871defc\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " Nov 03 18:21:31.518164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.518091 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\") pod \"4ad45e40-7c86-4b9e-919c-8b742871defc\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " Nov 03 18:21:31.518164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.518135 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\" (UniqueName: \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\") pod \"4ad45e40-7c86-4b9e-919c-8b742871defc\" (UID: \"4ad45e40-7c86-4b9e-919c-8b742871defc\") " Nov 03 18:21:31.518530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.518501 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca" pod "4ad45e40-7c86-4b9e-919c-8b742871defc" (UID: "4ad45e40-7c86-4b9e-919c-8b742871defc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:31.520406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.520381 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs" pod "4ad45e40-7c86-4b9e-919c-8b742871defc" (UID: "4ad45e40-7c86-4b9e-919c-8b742871defc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:21:31.520519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.520433 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm" pod "4ad45e40-7c86-4b9e-919c-8b742871defc" (UID: "4ad45e40-7c86-4b9e-919c-8b742871defc"). InnerVolumeSpecName "kube-api-access-mfdmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:31.520519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.520499 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls" pod "4ad45e40-7c86-4b9e-919c-8b742871defc" (UID: "4ad45e40-7c86-4b9e-919c-8b742871defc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:21:31.619611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.619478 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:31.619611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.619511 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4ad45e40-7c86-4b9e-919c-8b742871defc-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:31.619611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.619521 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ad45e40-7c86-4b9e-919c-8b742871defc-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:31.619611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.619531 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfdmm\" (UniqueName: \"kubernetes.io/projected/4ad45e40-7c86-4b9e-919c-8b742871defc-kube-api-access-mfdmm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:31.760080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.760043 4980 generic.go:356] "Generic (PLEG): container finished" podID="4ad45e40-7c86-4b9e-919c-8b742871defc" containerID="7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef" exitCode=0 Nov 03 18:21:31.760249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.760094 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" Nov 03 18:21:31.760249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.760109 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" event={"ID":"4ad45e40-7c86-4b9e-919c-8b742871defc","Type":"ContainerDied","Data":"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef"} Nov 03 18:21:31.760249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.760150 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24" event={"ID":"4ad45e40-7c86-4b9e-919c-8b742871defc","Type":"ContainerDied","Data":"5e0f3eb1eec0c4196a809b2a6cb038ac47df93c2db6b9cbbc085634a55a4961b"} Nov 03 18:21:31.760249 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.760172 4980 scope.go:122] "RemoveContainer" containerID="7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef" Nov 03 18:21:31.770323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.770299 4980 scope.go:122] "RemoveContainer" containerID="7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef" Nov 03 18:21:31.770670 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:31.770648 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef\": container with ID starting with 7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef not found: ID does not exist" containerID="7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef" Nov 03 18:21:31.770744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.770680 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef"} err="failed to get container status \"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef\": rpc error: code = NotFound desc = could not find container \"7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef\": container with ID starting with 7d55a721b66d014a36173328f847659004f724887c3aecf6c3267730f2da83ef not found: ID does not exist" Nov 03 18:21:31.928760 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.928679 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:21:31.934819 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:31.934793 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ca6b33-b5dbff4bb-lxb24"] Nov 03 18:21:32.293745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:32.293711 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ad45e40-7c86-4b9e-919c-8b742871defc" path="/var/lib/kubelet/pods/4ad45e40-7c86-4b9e-919c-8b742871defc/volumes" Nov 03 18:21:38.926839 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:38.926801 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:21:39.793485 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.793425 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:21:39.801755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.801727 4980 generic.go:356] "Generic (PLEG): container finished" podID="090d5579-9eed-4f9d-94d5-821b10da2536" containerID="c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7" exitCode=0 Nov 03 18:21:39.801892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.801770 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" Nov 03 18:21:39.801892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.801813 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" event={"ID":"090d5579-9eed-4f9d-94d5-821b10da2536","Type":"ContainerDied","Data":"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7"} Nov 03 18:21:39.801892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.801863 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v" event={"ID":"090d5579-9eed-4f9d-94d5-821b10da2536","Type":"ContainerDied","Data":"a288a50eb42d2fec2d00aee09b285352f940bd4ddd60904799a528fd9cd55b2c"} Nov 03 18:21:39.801892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.801886 4980 scope.go:122] "RemoveContainer" containerID="c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7" Nov 03 18:21:39.811134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.811109 4980 scope.go:122] "RemoveContainer" containerID="c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7" Nov 03 18:21:39.811429 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:39.811407 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7\": container with ID starting with c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7 not found: ID does not exist" containerID="c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7" Nov 03 18:21:39.811479 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.811439 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7"} err="failed to get container status \"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7\": rpc error: code = NotFound desc = could not find container \"c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7\": container with ID starting with c15fd9661028a66b31e8ff1d1bfc6712f9104eab402d46ad1ed98d4827ff35f7 not found: ID does not exist" Nov 03 18:21:39.912401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.912351 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\") pod \"090d5579-9eed-4f9d-94d5-821b10da2536\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " Nov 03 18:21:39.912675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.912653 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\" (UniqueName: \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\") pod \"090d5579-9eed-4f9d-94d5-821b10da2536\" (UID: \"090d5579-9eed-4f9d-94d5-821b10da2536\") " Nov 03 18:21:39.913134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.913025 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume" pod "090d5579-9eed-4f9d-94d5-821b10da2536" (UID: "090d5579-9eed-4f9d-94d5-821b10da2536"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:39.913438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.913402 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/090d5579-9eed-4f9d-94d5-821b10da2536-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:39.915960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:39.915933 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv" pod "090d5579-9eed-4f9d-94d5-821b10da2536" (UID: "090d5579-9eed-4f9d-94d5-821b10da2536"). InnerVolumeSpecName "kube-api-access-vktkv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:40.014493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:40.014459 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vktkv\" (UniqueName: \"kubernetes.io/projected/090d5579-9eed-4f9d-94d5-821b10da2536-kube-api-access-vktkv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:40.225648 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:40.225613 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:21:40.232197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:40.232166 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-ca6b33-855c5c86cf-wtl4v"] Nov 03 18:21:40.293484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:40.293450 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" path="/var/lib/kubelet/pods/090d5579-9eed-4f9d-94d5-821b10da2536/volumes" Nov 03 18:21:42.139659 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:42.139622 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:21:42.140061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:42.139857 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" containerID="cri-o://d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f" gracePeriod=30 Nov 03 18:21:44.520554 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:44.520511 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:21:47.401318 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.401294 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:21:47.491835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.491807 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.491989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.491842 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.491989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.491877 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.491989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.491918 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.491989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.491947 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.492197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492096 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.492197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492129 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.492197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492081 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:47.492197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492190 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.492425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492206 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:47.492425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492218 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\") pod \"a9b90180-6712-4f71-90a6-57098f5d95ac\" (UID: \"a9b90180-6712-4f71-90a6-57098f5d95ac\") " Nov 03 18:21:47.492425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492356 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:47.492620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492454 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:47.492620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492500 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:21:47.492882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492857 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.493141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492887 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.493141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492902 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.493141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492915 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.493141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.492930 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a9b90180-6712-4f71-90a6-57098f5d95ac-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.494386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.494359 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "kube-api-access-5htrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:47.494474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.494387 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:21:47.494474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.494395 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:21:47.494711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.494692 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token" pod "a9b90180-6712-4f71-90a6-57098f5d95ac" (UID: "a9b90180-6712-4f71-90a6-57098f5d95ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:21:47.593448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.593418 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.593448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.593447 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5htrp\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-kube-api-access-5htrp\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.593693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.593457 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.593693 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.593467 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a9b90180-6712-4f71-90a6-57098f5d95ac-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:21:47.856656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.856544 4980 generic.go:356] "Generic (PLEG): container finished" podID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerID="d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f" exitCode=0 Nov 03 18:21:47.856656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.856626 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" Nov 03 18:21:47.856656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.856635 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerDied","Data":"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f"} Nov 03 18:21:47.856951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.856699 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5" event={"ID":"a9b90180-6712-4f71-90a6-57098f5d95ac","Type":"ContainerDied","Data":"17d65d95d098fb1569bb2c3570f8aca4e53e6a354f497664f26b265a880c1d1a"} Nov 03 18:21:47.856951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.856722 4980 scope.go:122] "RemoveContainer" containerID="d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f" Nov 03 18:21:47.868823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.868806 4980 scope.go:122] "RemoveContainer" containerID="eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888" Nov 03 18:21:47.877101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.877082 4980 scope.go:122] "RemoveContainer" containerID="d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f" Nov 03 18:21:47.877379 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:47.877337 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f\": container with ID starting with d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f not found: ID does not exist" containerID="d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f" Nov 03 18:21:47.877379 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.877365 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f"} err="failed to get container status \"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f\": rpc error: code = NotFound desc = could not find container \"d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f\": container with ID starting with d02abdbc8b7c49a24817a88163c919aeebddd1fa4d5740cd4909382872ef1e9f not found: ID does not exist" Nov 03 18:21:47.877473 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.877384 4980 scope.go:122] "RemoveContainer" containerID="eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888" Nov 03 18:21:47.877636 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:21:47.877617 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888\": container with ID starting with eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888 not found: ID does not exist" containerID="eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888" Nov 03 18:21:47.877684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:47.877642 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888"} err="failed to get container status \"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888\": rpc error: code = NotFound desc = could not find container \"eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888\": container with ID starting with eb1e48f0e685e8a5a36f3ddeef41ae2473a02c45c50a6ffaca79d30b86db6888 not found: ID does not exist" Nov 03 18:21:48.025857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:48.025828 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:21:48.032337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:48.032312 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-ca6b33-openshift-default-5df86bc587-rndf5"] Nov 03 18:21:48.296777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:21:48.294714 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" path="/var/lib/kubelet/pods/a9b90180-6712-4f71-90a6-57098f5d95ac/volumes" Nov 03 18:22:13.788021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:13.785053 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:22:13.788021 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:13.785416 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" containerID="cri-o://996c50dcf9944f6b72a4a08d94b7d484266eb0b3734ed45f32db507d02024add" gracePeriod=30 Nov 03 18:22:14.520150 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:22:14.520109 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:22:17.076022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:17.075985 4980 patch_prober.go:29] interesting pod/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Nov 03 18:22:17.076436 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:17.076063 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Nov 03 18:22:19.010300 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.010270 4980 generic.go:356] "Generic (PLEG): container finished" podID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerID="996c50dcf9944f6b72a4a08d94b7d484266eb0b3734ed45f32db507d02024add" exitCode=0 Nov 03 18:22:19.010801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.010336 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerDied","Data":"996c50dcf9944f6b72a4a08d94b7d484266eb0b3734ed45f32db507d02024add"} Nov 03 18:22:19.010801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.010408 4980 scope.go:122] "RemoveContainer" containerID="368a4a0578f60464f70ce1b24e4a5fa774acc1b1d7a6f9f06613208f76d6ac77" Nov 03 18:22:19.060790 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.060732 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:22:19.103525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103495 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103545 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103618 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103638 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103660 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103692 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.103727 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103715 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.104082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103769 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.104082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103841 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\") pod \"0c515d25-41c3-4e44-9a64-954be42e00cb\" (UID: \"0c515d25-41c3-4e44-9a64-954be42e00cb\") " Nov 03 18:22:19.104082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.103866 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:19.104247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104071 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:22:19.104320 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104297 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:19.104483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104448 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:19.104548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104513 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:19.104548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104517 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.104691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.104557 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c515d25-41c3-4e44-9a64-954be42e00cb-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.106028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.106004 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:19.106232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.106211 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "kube-api-access-5jw4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:19.106433 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.106414 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:22:19.106521 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.106429 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token" pod "0c515d25-41c3-4e44-9a64-954be42e00cb" (UID: "0c515d25-41c3-4e44-9a64-954be42e00cb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:19.205915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205883 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.205915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205910 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.205915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205920 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.206155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205929 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.206155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205938 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jw4x\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-kube-api-access-5jw4x\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.206155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205947 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c515d25-41c3-4e44-9a64-954be42e00cb-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.206155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.205955 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c515d25-41c3-4e44-9a64-954be42e00cb-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:19.286852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.286812 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 18:22:19.705670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.705635 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:22:19.705869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:19.705844 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" containerID="cri-o://9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977" gracePeriod=30 Nov 03 18:22:20.017185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.017107 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" Nov 03 18:22:20.017185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.017114 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5" event={"ID":"0c515d25-41c3-4e44-9a64-954be42e00cb","Type":"ContainerDied","Data":"5eccd96a6502cf1490545234b6605f2c89ff35b59cd955cca5fb169495be3035"} Nov 03 18:22:20.017185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.017181 4980 scope.go:122] "RemoveContainer" containerID="996c50dcf9944f6b72a4a08d94b7d484266eb0b3734ed45f32db507d02024add" Nov 03 18:22:20.185843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.185759 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:22:20.191779 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.191749 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-0adf17-openshift-default-5f9d79764c-cxvv5"] Nov 03 18:22:20.292478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:20.292392 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" path="/var/lib/kubelet/pods/0c515d25-41c3-4e44-9a64-954be42e00cb/volumes" Nov 03 18:22:24.965695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:24.965667 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:22:25.043115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.043030 4980 generic.go:356] "Generic (PLEG): container finished" podID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerID="9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977" exitCode=0 Nov 03 18:22:25.043115 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.043079 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" Nov 03 18:22:25.043334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.043113 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" event={"ID":"a1c3f624-5986-47df-a72c-c0894ea403f9","Type":"ContainerDied","Data":"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977"} Nov 03 18:22:25.043334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.043174 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72" event={"ID":"a1c3f624-5986-47df-a72c-c0894ea403f9","Type":"ContainerDied","Data":"221b9ebce63bdf9085cce7226700f264f2642115f5daedcd889d81ce98de1aaf"} Nov 03 18:22:25.043334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.043197 4980 scope.go:122] "RemoveContainer" containerID="9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977" Nov 03 18:22:25.053386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.053369 4980 scope.go:122] "RemoveContainer" containerID="9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977" Nov 03 18:22:25.053758 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:22:25.053733 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977\": container with ID starting with 9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977 not found: ID does not exist" containerID="9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977" Nov 03 18:22:25.053854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.053771 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977"} err="failed to get container status \"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977\": rpc error: code = NotFound desc = could not find container \"9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977\": container with ID starting with 9516a0e20444080d15b3e791f42ab4b0336affb99983b74d5b79e2c5a58ba977 not found: ID does not exist" Nov 03 18:22:25.064015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.063995 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064023 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064055 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064086 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064203 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064249 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064382 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064275 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:25.064443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064388 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064502 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064509 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:25.064704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064555 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\") pod \"a1c3f624-5986-47df-a72c-c0894ea403f9\" (UID: \"a1c3f624-5986-47df-a72c-c0894ea403f9\") " Nov 03 18:22:25.064704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064624 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:22:25.064825 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064697 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:25.065025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.064885 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:25.065153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.065133 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a1c3f624-5986-47df-a72c-c0894ea403f9-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.065222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.065158 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.065222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.065175 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.065222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.065190 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.065222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.065204 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.066460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.066434 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:25.066603 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.066502 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "kube-api-access-c6nc7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:25.066689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.066662 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:25.066854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.066836 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo" pod "a1c3f624-5986-47df-a72c-c0894ea403f9" (UID: "a1c3f624-5986-47df-a72c-c0894ea403f9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:22:25.166475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.166442 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.166475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.166472 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6nc7\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-kube-api-access-c6nc7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.166475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.166484 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.166837 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.166492 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a1c3f624-5986-47df-a72c-c0894ea403f9-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:25.467677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.467640 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:22:25.477651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.477619 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-q9c72"] Nov 03 18:22:25.814386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814341 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c"] Nov 03 18:22:25.814764 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814751 4980 cpu_manager.go:414] "containerMap: removing container" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerName="istio-proxy" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814765 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerName="istio-proxy" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814775 4980 cpu_manager.go:414] "containerMap: removing container" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814780 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814797 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4a5f7daa-bc77-49d6-af16-52075beb0338" containerName="maas-api" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814802 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5f7daa-bc77-49d6-af16-52075beb0338" containerName="maas-api" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814812 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814817 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814825 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" Nov 03 18:22:25.814832 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814831 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814838 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814844 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814851 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6ecb4643-e2bf-4f7e-a110-f0927207496f" containerName="payload-processing" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814856 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ecb4643-e2bf-4f7e-a110-f0927207496f" containerName="payload-processing" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814871 4980 cpu_manager.go:414] "containerMap: removing container" podUID="4ad45e40-7c86-4b9e-919c-8b742871defc" containerName="maas-api" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814875 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ad45e40-7c86-4b9e-919c-8b742871defc" containerName="maas-api" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814888 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814893 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814951 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814960 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814966 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814971 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ecb4643-e2bf-4f7e-a110-f0927207496f" containerName="payload-processing" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814977 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1074adce-5d3a-4974-9ec1-c40f5abc4bbe" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814983 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ad45e40-7c86-4b9e-919c-8b742871defc" containerName="maas-api" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814993 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="61cf8d9c-af3f-4c06-afbf-862c223c23d9" containerName="istio-proxy" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.814997 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="090d5579-9eed-4f9d-94d5-821b10da2536" containerName="payload-pre-processing" Nov 03 18:22:25.815177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.815004 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5f7daa-bc77-49d6-af16-52075beb0338" containerName="maas-api" Nov 03 18:22:25.819142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.819126 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.872328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.872295 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:22:25.873705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.873677 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.873855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.873740 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rlrv\" (UniqueName: \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.873937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.873869 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.875252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.875226 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:22:25.878530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.878494 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c"] Nov 03 18:22:25.975118 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.975075 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.975529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.975139 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rlrv\" (UniqueName: \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.975529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.975186 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.975807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.975786 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:25.977774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:25.977754 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:26.038071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:26.038041 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rlrv\" (UniqueName: \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\") pod \"maas-api-revoke-sub-79986443-hxp9c\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:26.132544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:26.132470 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:26.292595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:26.292546 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1c3f624-5986-47df-a72c-c0894ea403f9" path="/var/lib/kubelet/pods/a1c3f624-5986-47df-a72c-c0894ea403f9/volumes" Nov 03 18:22:26.513815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:26.513787 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c"] Nov 03 18:22:26.516202 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:22:26.516163 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08ecd753_1a1f_433d_9bbf_81fb884a4ce2.slice/crio-0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad WatchSource:0}: Error finding container 0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad: Status 404 returned error can't find the container with id 0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad Nov 03 18:22:27.055449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:27.055357 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" event={"ID":"08ecd753-1a1f-433d-9bbf-81fb884a4ce2","Type":"ContainerStarted","Data":"19ed05017a0229530ce876ba257f11079501c58a7645dcb97d343275b5060fc5"} Nov 03 18:22:27.055449 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:27.055410 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" event={"ID":"08ecd753-1a1f-433d-9bbf-81fb884a4ce2","Type":"ContainerStarted","Data":"0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad"} Nov 03 18:22:27.175426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:27.175366 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" podStartSLOduration=2.17535071 podStartE2EDuration="2.17535071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:22:27.173688363 +0000 UTC m=+2927.002771967" watchObservedRunningTime="2026-11-03 18:22:27.17535071 +0000 UTC m=+2927.004434311" Nov 03 18:22:28.063957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:28.063920 4980 generic.go:356] "Generic (PLEG): container finished" podID="08ecd753-1a1f-433d-9bbf-81fb884a4ce2" containerID="19ed05017a0229530ce876ba257f11079501c58a7645dcb97d343275b5060fc5" exitCode=0 Nov 03 18:22:28.063957 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:28.063954 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" event={"ID":"08ecd753-1a1f-433d-9bbf-81fb884a4ce2","Type":"ContainerDied","Data":"19ed05017a0229530ce876ba257f11079501c58a7645dcb97d343275b5060fc5"} Nov 03 18:22:29.213453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.213427 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:29.308243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.308205 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\") pod \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " Nov 03 18:22:29.308419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.308359 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\") pod \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " Nov 03 18:22:29.308419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.308397 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\" (UniqueName: \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\") pod \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\" (UID: \"08ecd753-1a1f-433d-9bbf-81fb884a4ce2\") " Nov 03 18:22:29.308627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.308605 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca" pod "08ecd753-1a1f-433d-9bbf-81fb884a4ce2" (UID: "08ecd753-1a1f-433d-9bbf-81fb884a4ce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:22:29.308750 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.308734 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:29.310732 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.310703 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca" pod "08ecd753-1a1f-433d-9bbf-81fb884a4ce2" (UID: "08ecd753-1a1f-433d-9bbf-81fb884a4ce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:29.310849 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.310754 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv" pod "08ecd753-1a1f-433d-9bbf-81fb884a4ce2" (UID: "08ecd753-1a1f-433d-9bbf-81fb884a4ce2"). InnerVolumeSpecName "kube-api-access-4rlrv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:29.409766 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.409675 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:29.409766 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:29.409713 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rlrv\" (UniqueName: \"kubernetes.io/projected/08ecd753-1a1f-433d-9bbf-81fb884a4ce2-kube-api-access-4rlrv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:30.087821 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:30.087791 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" Nov 03 18:22:30.088007 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:30.087782 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c" event={"ID":"08ecd753-1a1f-433d-9bbf-81fb884a4ce2","Type":"ContainerDied","Data":"0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad"} Nov 03 18:22:30.088007 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:30.087963 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f6a9acfc041f2591c4e2435b6c5b27aeeeb446e45a5d81bd1e7bf1d96c009ad" Nov 03 18:22:34.307526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.307485 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" containerID="cri-o://1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63" gracePeriod=45 Nov 03 18:22:34.352096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.352057 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" probeResult="failure" output="Get \"https://10.132.0.57:8000/ready\": dial tcp 10.132.0.57:8000: connect: connection refused" Nov 03 18:22:34.560538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.560470 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:22:34.669559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669519 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669610 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\" (UniqueName: \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669679 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669781 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669708 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\" (UniqueName: \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669787 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669824 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:34.669978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669839 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\") pod \"afee383f-b486-4de5-a58f-a30e14a768ee\" (UID: \"afee383f-b486-4de5-a58f-a30e14a768ee\") " Nov 03 18:22:34.669978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.669909 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:34.670179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.670036 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:34.670253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.670230 4980 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-tmp-dir\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:34.670301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.670261 4980 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-home\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:34.670301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.670274 4980 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-model-cache\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:34.672219 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.672196 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:22:34.672219 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.672202 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:34.672361 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.672319 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m" pod "afee383f-b486-4de5-a58f-a30e14a768ee" (UID: "afee383f-b486-4de5-a58f-a30e14a768ee"). InnerVolumeSpecName "kube-api-access-mf87m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:34.772025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.771987 4980 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afee383f-b486-4de5-a58f-a30e14a768ee-dshm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:34.772025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.772028 4980 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afee383f-b486-4de5-a58f-a30e14a768ee-tls-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:34.772234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:34.772043 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf87m\" (UniqueName: \"kubernetes.io/projected/afee383f-b486-4de5-a58f-a30e14a768ee-kube-api-access-mf87m\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:35.112581 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.112537 4980 generic.go:356] "Generic (PLEG): container finished" podID="afee383f-b486-4de5-a58f-a30e14a768ee" containerID="1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63" exitCode=0 Nov 03 18:22:35.112762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.112598 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" Nov 03 18:22:35.112762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.112600 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" event={"ID":"afee383f-b486-4de5-a58f-a30e14a768ee","Type":"ContainerDied","Data":"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63"} Nov 03 18:22:35.112762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.112710 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c" event={"ID":"afee383f-b486-4de5-a58f-a30e14a768ee","Type":"ContainerDied","Data":"6c27c24c7ce9fc1ccfa4441c337dc4a1e942b636d32b2da32a3ba062de53a3d5"} Nov 03 18:22:35.112762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.112728 4980 scope.go:122] "RemoveContainer" containerID="1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63" Nov 03 18:22:35.122763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.122745 4980 scope.go:122] "RemoveContainer" containerID="1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63" Nov 03 18:22:35.123064 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:22:35.123044 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63\": container with ID starting with 1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63 not found: ID does not exist" containerID="1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63" Nov 03 18:22:35.123130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.123073 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63"} err="failed to get container status \"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63\": rpc error: code = NotFound desc = could not find container \"1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63\": container with ID starting with 1df1dfd6974fe4d94a93c71d095008f11f164c807cb7fc46ab5eeeb252413b63 not found: ID does not exist" Nov 03 18:22:35.283775 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.283740 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 18:22:35.333823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:35.333784 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-776ba7/premium-w3-776ba7-kserve-5c68559945-jdv9c"] Nov 03 18:22:36.293355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:36.293320 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" path="/var/lib/kubelet/pods/afee383f-b486-4de5-a58f-a30e14a768ee/volumes" Nov 03 18:22:40.444690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.444652 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 18:22:40.445200 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.444862 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" podUID="b46938a7-f0fe-4d64-9f98-04b73326b51c" containerName="maas-api" containerID="cri-o://eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd" gracePeriod=30 Nov 03 18:22:40.721111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.721085 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 18:22:40.831710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.831675 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") pod \"b46938a7-f0fe-4d64-9f98-04b73326b51c\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " Nov 03 18:22:40.832016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.831979 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\") pod \"b46938a7-f0fe-4d64-9f98-04b73326b51c\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " Nov 03 18:22:40.832157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.832024 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\" (UniqueName: \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\") pod \"b46938a7-f0fe-4d64-9f98-04b73326b51c\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " Nov 03 18:22:40.832157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.832068 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\") pod \"b46938a7-f0fe-4d64-9f98-04b73326b51c\" (UID: \"b46938a7-f0fe-4d64-9f98-04b73326b51c\") " Nov 03 18:22:40.832961 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.832928 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca" pod "b46938a7-f0fe-4d64-9f98-04b73326b51c" (UID: "b46938a7-f0fe-4d64-9f98-04b73326b51c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:22:40.836056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.836025 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5" pod "b46938a7-f0fe-4d64-9f98-04b73326b51c" (UID: "b46938a7-f0fe-4d64-9f98-04b73326b51c"). InnerVolumeSpecName "kube-api-access-d7fq5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:40.836263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.836226 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs" pod "b46938a7-f0fe-4d64-9f98-04b73326b51c" (UID: "b46938a7-f0fe-4d64-9f98-04b73326b51c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:40.836400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.836383 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls" pod "b46938a7-f0fe-4d64-9f98-04b73326b51c" (UID: "b46938a7-f0fe-4d64-9f98-04b73326b51c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:40.933527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.933497 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:40.933527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.933524 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b46938a7-f0fe-4d64-9f98-04b73326b51c-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:40.933527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.933534 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7fq5\" (UniqueName: \"kubernetes.io/projected/b46938a7-f0fe-4d64-9f98-04b73326b51c-kube-api-access-d7fq5\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:40.933789 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:40.933546 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b46938a7-f0fe-4d64-9f98-04b73326b51c-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:41.145116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.145034 4980 generic.go:356] "Generic (PLEG): container finished" podID="b46938a7-f0fe-4d64-9f98-04b73326b51c" containerID="eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd" exitCode=0 Nov 03 18:22:41.145116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.145100 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" Nov 03 18:22:41.145312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.145124 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" event={"ID":"b46938a7-f0fe-4d64-9f98-04b73326b51c","Type":"ContainerDied","Data":"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd"} Nov 03 18:22:41.145312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.145178 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4" event={"ID":"b46938a7-f0fe-4d64-9f98-04b73326b51c","Type":"ContainerDied","Data":"37cfd06fc250d98fa41a1cf84e23768f348d729802cdb00b2475a76b3ea0f94a"} Nov 03 18:22:41.145312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.145202 4980 scope.go:122] "RemoveContainer" containerID="eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd" Nov 03 18:22:41.154807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.154781 4980 scope.go:122] "RemoveContainer" containerID="eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd" Nov 03 18:22:41.155137 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:22:41.155112 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd\": container with ID starting with eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd not found: ID does not exist" containerID="eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd" Nov 03 18:22:41.155202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.155150 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd"} err="failed to get container status \"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd\": rpc error: code = NotFound desc = could not find container \"eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd\": container with ID starting with eb9998b6074125c43aaca00050417792c02430195140d317a46c06cfd05c69dd not found: ID does not exist" Nov 03 18:22:41.316020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.315528 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 18:22:41.322680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:41.322639 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w3-776ba7-cd9956-gwqt4"] Nov 03 18:22:42.292729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:42.292693 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b46938a7-f0fe-4d64-9f98-04b73326b51c" path="/var/lib/kubelet/pods/b46938a7-f0fe-4d64-9f98-04b73326b51c/volumes" Nov 03 18:22:44.519764 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:22:44.519727 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:22:56.971316 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:56.971281 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 18:22:56.973040 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:56.972998 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" podUID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" containerName="manager" containerID="cri-o://d3a15d1b175a3a7f9b0288f0755b73cf5fd2410255dac4dbeb763628d585d955" gracePeriod=30 Nov 03 18:22:57.225931 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.225857 4980 generic.go:356] "Generic (PLEG): container finished" podID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" containerID="d3a15d1b175a3a7f9b0288f0755b73cf5fd2410255dac4dbeb763628d585d955" exitCode=0 Nov 03 18:22:57.226051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.225932 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" event={"ID":"28ca8e1d-78fd-4787-bb84-d3f57ffdce03","Type":"ContainerDied","Data":"d3a15d1b175a3a7f9b0288f0755b73cf5fd2410255dac4dbeb763628d585d955"} Nov 03 18:22:57.226051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.225997 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" event={"ID":"28ca8e1d-78fd-4787-bb84-d3f57ffdce03","Type":"ContainerDied","Data":"5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b"} Nov 03 18:22:57.226051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.226040 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5493452f2a06bc7cd5fdfda11f05373111644b11e8e39fae94583c8b0e315c1b" Nov 03 18:22:57.234227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.234207 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 18:22:57.389418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.389384 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\") pod \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " Nov 03 18:22:57.389418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.389418 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\") pod \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " Nov 03 18:22:57.389680 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.389535 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\" (UniqueName: \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\") pod \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\" (UID: \"28ca8e1d-78fd-4787-bb84-d3f57ffdce03\") " Nov 03 18:22:57.391852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.391818 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46" pod "28ca8e1d-78fd-4787-bb84-d3f57ffdce03" (UID: "28ca8e1d-78fd-4787-bb84-d3f57ffdce03"). InnerVolumeSpecName "kube-api-access-2qs46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:22:57.391984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.391910 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs" pod "28ca8e1d-78fd-4787-bb84-d3f57ffdce03" (UID: "28ca8e1d-78fd-4787-bb84-d3f57ffdce03"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:57.392226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.392207 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert" pod "28ca8e1d-78fd-4787-bb84-d3f57ffdce03" (UID: "28ca8e1d-78fd-4787-bb84-d3f57ffdce03"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:22:57.491257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.491181 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qs46\" (UniqueName: \"kubernetes.io/projected/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-kube-api-access-2qs46\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:57.491257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.491211 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:57.491257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:57.491222 4980 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/28ca8e1d-78fd-4787-bb84-d3f57ffdce03-webhook-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:22:58.231413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:58.231381 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-kzlrh" Nov 03 18:22:58.403080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:58.403046 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 18:22:58.452997 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:58.452971 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-kzlrh"] Nov 03 18:22:59.408695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.408654 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:22:59.409312 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409289 4980 cpu_manager.go:414] "containerMap: removing container" podUID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" containerName="manager" Nov 03 18:22:59.409366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409318 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" containerName="manager" Nov 03 18:22:59.409366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409348 4980 cpu_manager.go:414] "containerMap: removing container" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" Nov 03 18:22:59.409366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409356 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409380 4980 cpu_manager.go:414] "containerMap: removing container" podUID="b46938a7-f0fe-4d64-9f98-04b73326b51c" containerName="maas-api" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409389 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b46938a7-f0fe-4d64-9f98-04b73326b51c" containerName="maas-api" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409401 4980 cpu_manager.go:414] "containerMap: removing container" podUID="08ecd753-1a1f-433d-9bbf-81fb884a4ce2" containerName="revoke-keys" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409410 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ecd753-1a1f-433d-9bbf-81fb884a4ce2" containerName="revoke-keys" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409438 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:59.409468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409446 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409561 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c515d25-41c3-4e44-9a64-954be42e00cb" containerName="istio-proxy" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409595 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b90180-6712-4f71-90a6-57098f5d95ac" containerName="istio-proxy" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409608 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" containerName="manager" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409621 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="afee383f-b486-4de5-a58f-a30e14a768ee" containerName="main" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409633 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="b46938a7-f0fe-4d64-9f98-04b73326b51c" containerName="maas-api" Nov 03 18:22:59.409697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.409642 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ecd753-1a1f-433d-9bbf-81fb884a4ce2" containerName="revoke-keys" Nov 03 18:22:59.413276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.413257 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.467513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.467481 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 18:22:59.513256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.513221 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kpr8\" (UniqueName: \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.513462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.513303 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.513462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.513428 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.516959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.516932 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 18:22:59.517106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.517002 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 18:22:59.517106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.517029 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bccd\"" Nov 03 18:22:59.519243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.519225 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:22:59.614138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.614094 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.614357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.614165 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kpr8\" (UniqueName: \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.614357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.614229 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.617078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.617047 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.617261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.617169 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.680285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.680214 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kpr8\" (UniqueName: \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\") pod \"maas-controller-b7df46f9c-c9mmw\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:22:59.727753 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:22:59.727449 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:23:00.143462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:00.143422 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:23:00.176630 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:23:00.175932 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d9b9093_5943_4788_a8b8_fde058cb8d37.slice/crio-08c43ae567c5aece7880fff4674f660239fe03147f97e449edff55fe78ec7f73 WatchSource:0}: Error finding container 08c43ae567c5aece7880fff4674f660239fe03147f97e449edff55fe78ec7f73: Status 404 returned error can't find the container with id 08c43ae567c5aece7880fff4674f660239fe03147f97e449edff55fe78ec7f73 Nov 03 18:23:00.178211 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:00.178190 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:23:00.242918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:00.242888 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" event={"ID":"5d9b9093-5943-4788-a8b8-fde058cb8d37","Type":"ContainerStarted","Data":"08c43ae567c5aece7880fff4674f660239fe03147f97e449edff55fe78ec7f73"} Nov 03 18:23:00.296785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:00.296748 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28ca8e1d-78fd-4787-bb84-d3f57ffdce03" path="/var/lib/kubelet/pods/28ca8e1d-78fd-4787-bb84-d3f57ffdce03/volumes" Nov 03 18:23:01.250154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:01.250109 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" event={"ID":"5d9b9093-5943-4788-a8b8-fde058cb8d37","Type":"ContainerStarted","Data":"cbcd2723e0e23a15440eb0a9d283dcb3e30e66f729b31c8e7c901633f38f3790"} Nov 03 18:23:01.250661 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:01.250170 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:23:01.380669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:01.378815 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" podStartSLOduration=1.433586748 podStartE2EDuration="2.378796056s" totalImagesPullingTime="945.209308ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:22:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:01.377175949 +0000 UTC m=+2961.206259552" watchObservedRunningTime="2026-11-03 18:23:01.378796056 +0000 UTC m=+2961.207879659" Nov 03 18:23:12.257045 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:12.257005 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:23:14.520258 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:23:14.520217 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:23:20.237979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.237926 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5"] Nov 03 18:23:20.241828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.241802 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.322645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.322608 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.322873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.322673 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.322873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.322746 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4xqw\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-kube-api-access-j4xqw\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.322873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.322815 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.323056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.322908 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.323056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.323003 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.323138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.323053 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.323138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.323123 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.323210 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.323159 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.424744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.424707 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.424955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.424768 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.424955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.424800 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4xqw\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-kube-api-access-j4xqw\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.424955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.424872 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.424955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.424938 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425008 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425046 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425089 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425124 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425354 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425409 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425396 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425695 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.425893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.425855 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.426084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.426064 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.427907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.427870 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.428129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.428109 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.459913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.459866 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5"] Nov 03 18:23:20.488299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.488227 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4xqw\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-kube-api-access-j4xqw\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.488299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.488236 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c53d3b05-4c4c-4c94-b156-684fd3e7435a-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-tcgv5\" (UID: \"c53d3b05-4c4c-4c94-b156-684fd3e7435a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.555562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.555525 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:20.581803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.581769 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25"] Nov 03 18:23:20.585463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.585442 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.641093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.641059 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:23:20.641280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.641132 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:23:20.645491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.645451 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25"] Nov 03 18:23:20.729167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.729126 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.729353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.729259 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.729353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.729288 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.830678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.830593 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.830678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.830642 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.830894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.830710 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.831450 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.831415 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.833178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.833154 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.894666 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.894633 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\") pod \"maas-api-revoke-sub-85237e45-rdf25\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.900034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.900006 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:20.991479 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.991451 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5"] Nov 03 18:23:20.994948 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:23:20.994917 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc53d3b05_4c4c_4c94_b156_684fd3e7435a.slice/crio-c9047a4a3e5bcbb8d9ad8b5d9500b1e044a22f6e60711135ff982a55aab2c323 WatchSource:0}: Error finding container c9047a4a3e5bcbb8d9ad8b5d9500b1e044a22f6e60711135ff982a55aab2c323: Status 404 returned error can't find the container with id c9047a4a3e5bcbb8d9ad8b5d9500b1e044a22f6e60711135ff982a55aab2c323 Nov 03 18:23:20.997217 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.997183 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:20.997314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.997249 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:20.997314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:20.997281 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:21.278602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.278556 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25"] Nov 03 18:23:21.281969 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:23:21.281938 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8382d9_4fb4_40a3_aeff_6bbf1651cc9b.slice/crio-947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944 WatchSource:0}: Error finding container 947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944: Status 404 returned error can't find the container with id 947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944 Nov 03 18:23:21.348846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.348803 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" event={"ID":"c53d3b05-4c4c-4c94-b156-684fd3e7435a","Type":"ContainerStarted","Data":"69d9e63463b3405652faeedd2b0910fb44845029fea71817147b634235b827c6"} Nov 03 18:23:21.349047 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.348876 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" event={"ID":"c53d3b05-4c4c-4c94-b156-684fd3e7435a","Type":"ContainerStarted","Data":"c9047a4a3e5bcbb8d9ad8b5d9500b1e044a22f6e60711135ff982a55aab2c323"} Nov 03 18:23:21.350194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.350159 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" event={"ID":"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b","Type":"ContainerStarted","Data":"947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944"} Nov 03 18:23:21.472389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.472333 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podStartSLOduration=1.472319389 podStartE2EDuration="1.472319389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:21.471014544 +0000 UTC m=+2981.300098146" watchObservedRunningTime="2026-11-03 18:23:21.472319389 +0000 UTC m=+2981.301402992" Nov 03 18:23:21.571075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:21.570954 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:22.381756 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:22.381709 4980 generic.go:356] "Generic (PLEG): container finished" podID="9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" containerID="4de0bd4100b349eaa41de815b5743116186614c38e25d6c72fb28d87f968256a" exitCode=0 Nov 03 18:23:22.382198 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:22.381776 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" event={"ID":"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b","Type":"ContainerDied","Data":"4de0bd4100b349eaa41de815b5743116186614c38e25d6c72fb28d87f968256a"} Nov 03 18:23:22.577667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:22.576563 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:22.577667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:22.576678 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:23.544127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.544101 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:23.580014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.579966 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:23.580194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.580032 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:23.683050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.682914 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\") pod \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " Nov 03 18:23:23.683378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.683350 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca" pod "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" (UID: "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:23:23.683663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.683638 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\") pod \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " Nov 03 18:23:23.683847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.683828 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\") pod \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\" (UID: \"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b\") " Nov 03 18:23:23.686372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.686323 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j" pod "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" (UID: "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b"). InnerVolumeSpecName "kube-api-access-scz5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:23:23.686542 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.686520 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca" pod "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" (UID: "9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:23.784915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.784881 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:23.784915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.784911 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:23.784915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:23.784922 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-scz5j\" (UniqueName: \"kubernetes.io/projected/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b-kube-api-access-scz5j\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:24.392822 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:24.392792 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" event={"ID":"9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b","Type":"ContainerDied","Data":"947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944"} Nov 03 18:23:24.392993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:24.392837 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="947a8d9b4afa7ede7734992cc8a159377059e6dda90b7d28c416997949115944" Nov 03 18:23:24.392993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:24.392835 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25" Nov 03 18:23:24.575654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:24.568943 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:24.575654 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:24.569004 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:25.564600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:25.562069 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:25.564600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:25.562144 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:26.573596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:26.566914 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:26.573596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:26.566977 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:27.556105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:27.556075 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:27.556284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:27.556127 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:28.556660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:28.556628 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" start-of-body= Nov 03 18:23:28.557177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:28.556688 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": dial tcp 10.132.0.169:15021: connect: connection refused" Nov 03 18:23:30.598740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:30.598694 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:23:30.599293 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:30.598768 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:23:31.610722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:31.610685 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-tcgv5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:23:31.611224 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:31.610748 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" podUID="c53d3b05-4c4c-4c94-b156-684fd3e7435a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:23:31.627560 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:31.627530 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:31.628566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:31.628547 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:31.629164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:31.629148 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-tcgv5" Nov 03 18:23:33.316345 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.316302 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:23:33.316940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.316563 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" podUID="5d9b9093-5943-4788-a8b8-fde058cb8d37" containerName="manager" containerID="cri-o://cbcd2723e0e23a15440eb0a9d283dcb3e30e66f729b31c8e7c901633f38f3790" gracePeriod=30 Nov 03 18:23:33.438046 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.438021 4980 generic.go:356] "Generic (PLEG): container finished" podID="5d9b9093-5943-4788-a8b8-fde058cb8d37" containerID="cbcd2723e0e23a15440eb0a9d283dcb3e30e66f729b31c8e7c901633f38f3790" exitCode=0 Nov 03 18:23:33.438141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.438095 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" event={"ID":"5d9b9093-5943-4788-a8b8-fde058cb8d37","Type":"ContainerDied","Data":"cbcd2723e0e23a15440eb0a9d283dcb3e30e66f729b31c8e7c901633f38f3790"} Nov 03 18:23:33.574237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.574176 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:23:33.671545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.671513 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\" (UniqueName: \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\") pod \"5d9b9093-5943-4788-a8b8-fde058cb8d37\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " Nov 03 18:23:33.671712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.671557 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\") pod \"5d9b9093-5943-4788-a8b8-fde058cb8d37\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " Nov 03 18:23:33.673826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.673792 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8" pod "5d9b9093-5943-4788-a8b8-fde058cb8d37" (UID: "5d9b9093-5943-4788-a8b8-fde058cb8d37"). InnerVolumeSpecName "kube-api-access-2kpr8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:23:33.673826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.673815 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs" pod "5d9b9093-5943-4788-a8b8-fde058cb8d37" (UID: "5d9b9093-5943-4788-a8b8-fde058cb8d37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:33.772620 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.772564 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\") pod \"5d9b9093-5943-4788-a8b8-fde058cb8d37\" (UID: \"5d9b9093-5943-4788-a8b8-fde058cb8d37\") " Nov 03 18:23:33.772911 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.772891 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kpr8\" (UniqueName: \"kubernetes.io/projected/5d9b9093-5943-4788-a8b8-fde058cb8d37-kube-api-access-2kpr8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:33.773001 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.772915 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:33.774857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.774829 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert" pod "5d9b9093-5943-4788-a8b8-fde058cb8d37" (UID: "5d9b9093-5943-4788-a8b8-fde058cb8d37"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:23:33.873692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:33.873546 4980 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5d9b9093-5943-4788-a8b8-fde058cb8d37-webhook-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:23:34.444672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:34.444642 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" Nov 03 18:23:34.445206 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:34.444669 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-c9mmw" event={"ID":"5d9b9093-5943-4788-a8b8-fde058cb8d37","Type":"ContainerDied","Data":"08c43ae567c5aece7880fff4674f660239fe03147f97e449edff55fe78ec7f73"} Nov 03 18:23:34.445206 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:34.444730 4980 scope.go:122] "RemoveContainer" containerID="cbcd2723e0e23a15440eb0a9d283dcb3e30e66f729b31c8e7c901633f38f3790" Nov 03 18:23:34.614222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:34.614150 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:23:34.664303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:34.664278 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-c9mmw"] Nov 03 18:23:35.360538 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.360499 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk"] Nov 03 18:23:35.361006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.360985 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5d9b9093-5943-4788-a8b8-fde058cb8d37" containerName="manager" Nov 03 18:23:35.361101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.361008 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d9b9093-5943-4788-a8b8-fde058cb8d37" containerName="manager" Nov 03 18:23:35.361101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.361020 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" containerName="revoke-keys" Nov 03 18:23:35.361101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.361027 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" containerName="revoke-keys" Nov 03 18:23:35.361255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.361148 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d9b9093-5943-4788-a8b8-fde058cb8d37" containerName="manager" Nov 03 18:23:35.361255 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.361162 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" containerName="revoke-keys" Nov 03 18:23:35.365529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.365507 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385290 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385431 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385328 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385431 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385345 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385459 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385525 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c6c7af-e170-4045-bb87-4cca726a11f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385615 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsjcm\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-kube-api-access-qsjcm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385685 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385633 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385685 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385679 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.385783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.385736 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.468277 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.468244 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk"] Nov 03 18:23:35.486393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486361 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsjcm\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-kube-api-access-qsjcm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486405 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486480 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486506 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486558 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486537 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486814 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486564 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486814 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486728 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486814 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486795 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486892 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c6c7af-e170-4045-bb87-4cca726a11f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.486988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.486934 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.487104 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.487005 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-workload-certs\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.487561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.487536 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-data\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.487561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.487536 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-credential-socket\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.488028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.488002 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32c6c7af-e170-4045-bb87-4cca726a11f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.489103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.489083 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.489222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.489204 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.549664 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.549641 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-istio-token\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.549783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.549719 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsjcm\" (UniqueName: \"kubernetes.io/projected/32c6c7af-e170-4045-bb87-4cca726a11f7-kube-api-access-qsjcm\") pod \"maas-default-gateway-openshift-default-5df48fcc99-bftjk\" (UID: \"32c6c7af-e170-4045-bb87-4cca726a11f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:35.677964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:35.677898 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:36.111953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.111929 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk"] Nov 03 18:23:36.114304 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:23:36.114266 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c6c7af_e170_4045_bb87_4cca726a11f7.slice/crio-f2a51d40028020e5eea2a1ff3a9dca119341e37c9a7078b12388fd38031024c4 WatchSource:0}: Error finding container f2a51d40028020e5eea2a1ff3a9dca119341e37c9a7078b12388fd38031024c4: Status 404 returned error can't find the container with id f2a51d40028020e5eea2a1ff3a9dca119341e37c9a7078b12388fd38031024c4 Nov 03 18:23:36.116417 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.116386 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:36.116525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.116462 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:36.116525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.116505 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:23:36.166002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.165910 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-b7df46f9c-p85x2"] Nov 03 18:23:36.169450 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.169428 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.193561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.193525 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-webhook-cert\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.193758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.193650 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvp4l\" (UniqueName: \"kubernetes.io/projected/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-kube-api-access-jvp4l\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.193758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.193728 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-metrics-certs\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.223703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.223672 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Nov 03 18:23:36.224067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.224051 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Nov 03 18:23:36.272509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.272479 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2bccd\"" Nov 03 18:23:36.272509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.272480 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Nov 03 18:23:36.276111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.276085 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-p85x2"] Nov 03 18:23:36.294888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.294251 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-webhook-cert\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.294888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.294320 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvp4l\" (UniqueName: \"kubernetes.io/projected/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-kube-api-access-jvp4l\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.294888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.294455 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-metrics-certs\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.296453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.296424 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d9b9093-5943-4788-a8b8-fde058cb8d37" path="/var/lib/kubelet/pods/5d9b9093-5943-4788-a8b8-fde058cb8d37/volumes" Nov 03 18:23:36.297177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.297154 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-metrics-certs\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.297812 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.297789 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-webhook-cert\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.357706 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.357676 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvp4l\" (UniqueName: \"kubernetes.io/projected/fbfd47f6-aae9-4164-ae0b-6ed876a106ea-kube-api-access-jvp4l\") pod \"maas-controller-b7df46f9c-p85x2\" (UID: \"fbfd47f6-aae9-4164-ae0b-6ed876a106ea\") " pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.461626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.461564 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" event={"ID":"32c6c7af-e170-4045-bb87-4cca726a11f7","Type":"ContainerStarted","Data":"2648e4623d9baa05635c1672a5315f85eacb44acb59b64833def3bf90d5f44b7"} Nov 03 18:23:36.461804 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.461649 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" event={"ID":"32c6c7af-e170-4045-bb87-4cca726a11f7","Type":"ContainerStarted","Data":"f2a51d40028020e5eea2a1ff3a9dca119341e37c9a7078b12388fd38031024c4"} Nov 03 18:23:36.483944 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.483912 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:36.588619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.588530 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podStartSLOduration=1.588503405 podStartE2EDuration="1.588503405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:36.585501612 +0000 UTC m=+2996.414585214" watchObservedRunningTime="2026-11-03 18:23:36.588503405 +0000 UTC m=+2996.417587077" Nov 03 18:23:36.679599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.678697 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:36.880626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:36.876219 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-b7df46f9c-p85x2"] Nov 03 18:23:36.881360 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:23:36.881314 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfd47f6_aae9_4164_ae0b_6ed876a106ea.slice/crio-1eeb27cbf66651690273208e4223b2157e695354a93cbe5c2f451839bb74a2d3 WatchSource:0}: Error finding container 1eeb27cbf66651690273208e4223b2157e695354a93cbe5c2f451839bb74a2d3: Status 404 returned error can't find the container with id 1eeb27cbf66651690273208e4223b2157e695354a93cbe5c2f451839bb74a2d3 Nov 03 18:23:37.485239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:37.485197 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-p85x2" event={"ID":"fbfd47f6-aae9-4164-ae0b-6ed876a106ea","Type":"ContainerStarted","Data":"1eeb27cbf66651690273208e4223b2157e695354a93cbe5c2f451839bb74a2d3"} Nov 03 18:23:37.681541 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:37.681501 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:37.681738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:37.681563 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:38.490950 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:38.490913 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-b7df46f9c-p85x2" event={"ID":"fbfd47f6-aae9-4164-ae0b-6ed876a106ea","Type":"ContainerStarted","Data":"81550af6357bf69ace000b77cc8cc9bb1a5858f863a75e1b589cb85418954889"} Nov 03 18:23:38.491394 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:38.491150 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:23:38.612017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:38.611962 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-b7df46f9c-p85x2" podStartSLOduration=1.7663561209999998 podStartE2EDuration="2.611946949s" totalImagesPullingTime="845.590828ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:23:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:23:38.610834248 +0000 UTC m=+2998.439917853" watchObservedRunningTime="2026-11-03 18:23:38.611946949 +0000 UTC m=+2998.441030551" Nov 03 18:23:38.680052 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:38.680014 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:38.680231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:38.680070 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:39.679596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:39.678933 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:39.679596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:39.678995 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:40.679978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:40.679940 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:40.680406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:40.679999 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:41.136807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.136772 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:23:41.143719 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.143697 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:23:41.201510 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.201484 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:23:41.206362 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.206334 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:23:41.679599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.678945 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:41.679599 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:41.679012 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:42.560085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:42.560046 4980 scope.go:122] "RemoveContainer" containerID="d3a15d1b175a3a7f9b0288f0755b73cf5fd2410255dac4dbeb763628d585d955" Nov 03 18:23:42.678534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:42.678501 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:42.678713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:42.678561 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:43.678758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:43.678719 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" start-of-body= Nov 03 18:23:43.679248 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:43.678775 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": dial tcp 10.132.0.171:15021: connect: connection refused" Nov 03 18:23:44.522778 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:23:44.522742 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:23:44.684251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:44.684221 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:44.684729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:44.684600 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:44.685441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:44.685423 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" Nov 03 18:23:49.497973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:23:49.497940 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-b7df46f9c-p85x2" Nov 03 18:24:14.519399 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:24:14.519315 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:24:32.518350 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.518302 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7"] Nov 03 18:24:32.522256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.522237 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.585847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.585810 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzrkx\" (UniqueName: \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.586056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.585880 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.586056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.585965 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.625828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.625760 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:24:32.626071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.625803 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:24:32.628370 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.628343 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7"] Nov 03 18:24:32.687519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.687483 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzrkx\" (UniqueName: \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.687757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.687550 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.687757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.687621 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.688372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.688287 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.690171 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.690149 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.751422 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.751388 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzrkx\" (UniqueName: \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\") pod \"maas-api-revoke-sub-c1e271a9-ldrx7\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:32.834672 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:32.834552 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:33.212403 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:33.212376 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7"] Nov 03 18:24:33.214705 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:24:33.214675 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9963d6d8_3f02_4f7a_a3d0_ce226789db5d.slice/crio-a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a WatchSource:0}: Error finding container a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a: Status 404 returned error can't find the container with id a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a Nov 03 18:24:33.782261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:33.782169 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" event={"ID":"9963d6d8-3f02-4f7a-a3d0-ce226789db5d","Type":"ContainerStarted","Data":"3fd0c53b38a1e76f9eade44a3ce9d605455114b014682da7c196a8ffd6cafb2b"} Nov 03 18:24:33.782261 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:33.782230 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" event={"ID":"9963d6d8-3f02-4f7a-a3d0-ce226789db5d","Type":"ContainerStarted","Data":"a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a"} Nov 03 18:24:33.902656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:33.902594 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" podStartSLOduration=1.902556368 podStartE2EDuration="1.902556368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:24:33.900659679 +0000 UTC m=+3053.729743282" watchObservedRunningTime="2026-11-03 18:24:33.902556368 +0000 UTC m=+3053.731639972" Nov 03 18:24:34.788322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:34.788285 4980 generic.go:356] "Generic (PLEG): container finished" podID="9963d6d8-3f02-4f7a-a3d0-ce226789db5d" containerID="3fd0c53b38a1e76f9eade44a3ce9d605455114b014682da7c196a8ffd6cafb2b" exitCode=0 Nov 03 18:24:34.788752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:34.788369 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" event={"ID":"9963d6d8-3f02-4f7a-a3d0-ce226789db5d","Type":"ContainerDied","Data":"3fd0c53b38a1e76f9eade44a3ce9d605455114b014682da7c196a8ffd6cafb2b"} Nov 03 18:24:35.933090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:35.933062 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:36.024524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.024496 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\") pod \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " Nov 03 18:24:36.024704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.024533 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\") pod \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " Nov 03 18:24:36.024704 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.024556 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\" (UniqueName: \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\") pod \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\" (UID: \"9963d6d8-3f02-4f7a-a3d0-ce226789db5d\") " Nov 03 18:24:36.024859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.024840 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca" pod "9963d6d8-3f02-4f7a-a3d0-ce226789db5d" (UID: "9963d6d8-3f02-4f7a-a3d0-ce226789db5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:24:36.026835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.026804 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca" pod "9963d6d8-3f02-4f7a-a3d0-ce226789db5d" (UID: "9963d6d8-3f02-4f7a-a3d0-ce226789db5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:24:36.026937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.026810 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx" pod "9963d6d8-3f02-4f7a-a3d0-ce226789db5d" (UID: "9963d6d8-3f02-4f7a-a3d0-ce226789db5d"). InnerVolumeSpecName "kube-api-access-jzrkx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:24:36.125734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.125658 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:24:36.125734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.125684 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:24:36.125734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.125695 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzrkx\" (UniqueName: \"kubernetes.io/projected/9963d6d8-3f02-4f7a-a3d0-ce226789db5d-kube-api-access-jzrkx\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:24:36.799738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.799710 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" Nov 03 18:24:36.799738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.799743 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7" event={"ID":"9963d6d8-3f02-4f7a-a3d0-ce226789db5d","Type":"ContainerDied","Data":"a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a"} Nov 03 18:24:36.799981 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:24:36.799789 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a44d746cd64c9b7643fb75a1014d420ce82e572a47cef66b52ed6d761a26777a" Nov 03 18:24:44.518940 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:24:44.518900 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:25:03.997416 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:03.997373 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr"] Nov 03 18:25:03.998120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:03.997983 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9963d6d8-3f02-4f7a-a3d0-ce226789db5d" containerName="revoke-keys" Nov 03 18:25:03.998120 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:03.998003 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9963d6d8-3f02-4f7a-a3d0-ce226789db5d" containerName="revoke-keys" Nov 03 18:25:03.998274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:03.998157 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9963d6d8-3f02-4f7a-a3d0-ce226789db5d" containerName="revoke-keys" Nov 03 18:25:04.001871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.001848 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.056481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.056450 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:25:04.101359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.101323 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.101544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.101374 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dnh8\" (UniqueName: \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.101625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.101535 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.105662 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.105640 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:25:04.107669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.107650 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr"] Nov 03 18:25:04.202374 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.202334 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.202545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.202409 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.202545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.202447 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dnh8\" (UniqueName: \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.203129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.203101 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.204950 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.204935 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.265875 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.265807 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dnh8\" (UniqueName: \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\") pod \"maas-api-revoke-sub-2e1f5b5f-4pbkr\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.315357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.315331 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:04.696995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.696965 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr"] Nov 03 18:25:04.699537 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:25:04.699505 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc41dc66b_b16b_4ba9_bebe_dc191d06c746.slice/crio-4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6 WatchSource:0}: Error finding container 4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6: Status 404 returned error can't find the container with id 4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6 Nov 03 18:25:04.965629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:04.965598 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" event={"ID":"c41dc66b-b16b-4ba9-bebe-dc191d06c746","Type":"ContainerStarted","Data":"4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6"} Nov 03 18:25:05.971889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:05.971856 4980 generic.go:356] "Generic (PLEG): container finished" podID="c41dc66b-b16b-4ba9-bebe-dc191d06c746" containerID="cdf457e888d4ae3444739e80859dbe77773d4cd4305675a9fc5f108fa969d2d7" exitCode=0 Nov 03 18:25:05.972323 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:05.971941 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" event={"ID":"c41dc66b-b16b-4ba9-bebe-dc191d06c746","Type":"ContainerDied","Data":"cdf457e888d4ae3444739e80859dbe77773d4cd4305675a9fc5f108fa969d2d7"} Nov 03 18:25:07.117496 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.117465 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:07.228854 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.228777 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\") pod \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " Nov 03 18:25:07.228994 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.228886 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\") pod \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " Nov 03 18:25:07.229036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.229014 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\" (UniqueName: \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\") pod \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\" (UID: \"c41dc66b-b16b-4ba9-bebe-dc191d06c746\") " Nov 03 18:25:07.229245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.229223 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca" pod "c41dc66b-b16b-4ba9-bebe-dc191d06c746" (UID: "c41dc66b-b16b-4ba9-bebe-dc191d06c746"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:25:07.229313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.229290 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:07.231164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.231137 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8" pod "c41dc66b-b16b-4ba9-bebe-dc191d06c746" (UID: "c41dc66b-b16b-4ba9-bebe-dc191d06c746"). InnerVolumeSpecName "kube-api-access-4dnh8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:25:07.231164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.231154 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca" pod "c41dc66b-b16b-4ba9-bebe-dc191d06c746" (UID: "c41dc66b-b16b-4ba9-bebe-dc191d06c746"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:25:07.329816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.329781 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c41dc66b-b16b-4ba9-bebe-dc191d06c746-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:07.329816 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.329813 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dnh8\" (UniqueName: \"kubernetes.io/projected/c41dc66b-b16b-4ba9-bebe-dc191d06c746-kube-api-access-4dnh8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:07.983170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.983131 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" event={"ID":"c41dc66b-b16b-4ba9-bebe-dc191d06c746","Type":"ContainerDied","Data":"4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6"} Nov 03 18:25:07.983353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.983177 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr" Nov 03 18:25:07.983353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:07.983186 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4663a145186955b3e8c62b9bce36ed048ef3651b1c8eb070d5aef063203e56a6" Nov 03 18:25:14.519282 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:25:14.519237 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:25:16.483411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.483373 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx"] Nov 03 18:25:16.484038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.484016 4980 cpu_manager.go:414] "containerMap: removing container" podUID="c41dc66b-b16b-4ba9-bebe-dc191d06c746" containerName="revoke-keys" Nov 03 18:25:16.484136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.484041 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c41dc66b-b16b-4ba9-bebe-dc191d06c746" containerName="revoke-keys" Nov 03 18:25:16.484257 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.484183 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c41dc66b-b16b-4ba9-bebe-dc191d06c746" containerName="revoke-keys" Nov 03 18:25:16.487375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.487352 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.590874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.590846 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:25:16.590874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.590855 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:25:16.593041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.593021 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx"] Nov 03 18:25:16.616551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.616525 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dglfm\" (UniqueName: \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.616742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.616643 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.616742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.616687 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.717593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.717534 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.717757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.717646 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.717806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.717758 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dglfm\" (UniqueName: \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.718231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.718202 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.720109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.720083 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.780896 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.780834 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dglfm\" (UniqueName: \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\") pod \"maas-api-revoke-sub-ec843018-7vhnx\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:16.800313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:16.800287 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:17.187529 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:17.187503 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx"] Nov 03 18:25:17.190072 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:25:17.190042 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2680bc60_5cbb_4ebf_9388_574ee769124b.slice/crio-864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f WatchSource:0}: Error finding container 864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f: Status 404 returned error can't find the container with id 864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f Nov 03 18:25:18.032912 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:18.032874 4980 generic.go:356] "Generic (PLEG): container finished" podID="2680bc60-5cbb-4ebf-9388-574ee769124b" containerID="1080af2dd245835842d924517a9d7c2345ae9eb8437449d226d3a89e99b07f22" exitCode=0 Nov 03 18:25:18.033377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:18.032962 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" event={"ID":"2680bc60-5cbb-4ebf-9388-574ee769124b","Type":"ContainerDied","Data":"1080af2dd245835842d924517a9d7c2345ae9eb8437449d226d3a89e99b07f22"} Nov 03 18:25:18.033377 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:18.033043 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" event={"ID":"2680bc60-5cbb-4ebf-9388-574ee769124b","Type":"ContainerStarted","Data":"864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f"} Nov 03 18:25:19.187335 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.187311 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:19.241211 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.241189 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\") pod \"2680bc60-5cbb-4ebf-9388-574ee769124b\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " Nov 03 18:25:19.241336 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.241247 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\" (UniqueName: \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\") pod \"2680bc60-5cbb-4ebf-9388-574ee769124b\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " Nov 03 18:25:19.241393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.241373 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\") pod \"2680bc60-5cbb-4ebf-9388-574ee769124b\" (UID: \"2680bc60-5cbb-4ebf-9388-574ee769124b\") " Nov 03 18:25:19.241643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.241619 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca" pod "2680bc60-5cbb-4ebf-9388-574ee769124b" (UID: "2680bc60-5cbb-4ebf-9388-574ee769124b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:25:19.243593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.243545 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca" pod "2680bc60-5cbb-4ebf-9388-574ee769124b" (UID: "2680bc60-5cbb-4ebf-9388-574ee769124b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:25:19.243593 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.243553 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm" pod "2680bc60-5cbb-4ebf-9388-574ee769124b" (UID: "2680bc60-5cbb-4ebf-9388-574ee769124b"). InnerVolumeSpecName "kube-api-access-dglfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:25:19.342717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.342633 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:19.342717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.342658 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2680bc60-5cbb-4ebf-9388-574ee769124b-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:19.342717 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:19.342668 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dglfm\" (UniqueName: \"kubernetes.io/projected/2680bc60-5cbb-4ebf-9388-574ee769124b-kube-api-access-dglfm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:20.044008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:20.043977 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" Nov 03 18:25:20.044008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:20.043982 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx" event={"ID":"2680bc60-5cbb-4ebf-9388-574ee769124b","Type":"ContainerDied","Data":"864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f"} Nov 03 18:25:20.044232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:20.044043 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="864c0effa75358e90052261eedac59cd355d34d51b0eca2c9b4b595fd44a935f" Nov 03 18:25:44.519185 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:25:44.519149 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:25:51.891517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.891479 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr"] Nov 03 18:25:51.892007 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.891948 4980 cpu_manager.go:414] "containerMap: removing container" podUID="2680bc60-5cbb-4ebf-9388-574ee769124b" containerName="revoke-keys" Nov 03 18:25:51.892007 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.891960 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2680bc60-5cbb-4ebf-9388-574ee769124b" containerName="revoke-keys" Nov 03 18:25:51.892082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.892059 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="2680bc60-5cbb-4ebf-9388-574ee769124b" containerName="revoke-keys" Nov 03 18:25:51.895364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.895346 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:51.949761 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.949730 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:25:51.998711 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:51.998671 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:25:52.001076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.001053 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr"] Nov 03 18:25:52.006099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.006079 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.006172 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.006113 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5m98\" (UniqueName: \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.006172 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.006146 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.107675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.107642 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.107675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.107689 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5m98\" (UniqueName: \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.107947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.107729 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.108626 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.108602 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.110229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.110206 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.171039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.170982 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5m98\" (UniqueName: \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\") pod \"maas-api-revoke-sub-046e1343-cd9jr\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.208461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.208433 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:52.588034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:52.588011 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr"] Nov 03 18:25:52.590536 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:25:52.590506 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cf26149_7df2_4580_a396_d66c8934117a.slice/crio-b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d WatchSource:0}: Error finding container b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d: Status 404 returned error can't find the container with id b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d Nov 03 18:25:53.236224 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:53.236185 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" event={"ID":"6cf26149-7df2-4580-a396-d66c8934117a","Type":"ContainerStarted","Data":"78074ec859999ef3409dc9b2c80e118f4d3fb42859c8e32b3dea67edaacfb14a"} Nov 03 18:25:53.236722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:53.236260 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" event={"ID":"6cf26149-7df2-4580-a396-d66c8934117a","Type":"ContainerStarted","Data":"b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d"} Nov 03 18:25:53.362767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:53.362713 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" podStartSLOduration=2.362699015 podStartE2EDuration="2.362699015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:25:53.361728616 +0000 UTC m=+3133.190812219" watchObservedRunningTime="2026-11-03 18:25:53.362699015 +0000 UTC m=+3133.191782617" Nov 03 18:25:54.243829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:54.243791 4980 generic.go:356] "Generic (PLEG): container finished" podID="6cf26149-7df2-4580-a396-d66c8934117a" containerID="78074ec859999ef3409dc9b2c80e118f4d3fb42859c8e32b3dea67edaacfb14a" exitCode=0 Nov 03 18:25:54.244272 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:54.243878 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" event={"ID":"6cf26149-7df2-4580-a396-d66c8934117a","Type":"ContainerDied","Data":"78074ec859999ef3409dc9b2c80e118f4d3fb42859c8e32b3dea67edaacfb14a"} Nov 03 18:25:55.412801 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.412774 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:55.446948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.446919 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\") pod \"6cf26149-7df2-4580-a396-d66c8934117a\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " Nov 03 18:25:55.447090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.446975 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\" (UniqueName: \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\") pod \"6cf26149-7df2-4580-a396-d66c8934117a\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " Nov 03 18:25:55.447149 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.447102 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\") pod \"6cf26149-7df2-4580-a396-d66c8934117a\" (UID: \"6cf26149-7df2-4580-a396-d66c8934117a\") " Nov 03 18:25:55.447481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.447456 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca" pod "6cf26149-7df2-4580-a396-d66c8934117a" (UID: "6cf26149-7df2-4580-a396-d66c8934117a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:25:55.449191 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.449166 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98" pod "6cf26149-7df2-4580-a396-d66c8934117a" (UID: "6cf26149-7df2-4580-a396-d66c8934117a"). InnerVolumeSpecName "kube-api-access-b5m98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:25:55.449256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.449169 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca" pod "6cf26149-7df2-4580-a396-d66c8934117a" (UID: "6cf26149-7df2-4580-a396-d66c8934117a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:25:55.548942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.548868 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:55.548942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.548898 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5m98\" (UniqueName: \"kubernetes.io/projected/6cf26149-7df2-4580-a396-d66c8934117a-kube-api-access-b5m98\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:55.548942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:55.548909 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cf26149-7df2-4580-a396-d66c8934117a-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:25:56.256099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:56.256069 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" Nov 03 18:25:56.256285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:56.256102 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr" event={"ID":"6cf26149-7df2-4580-a396-d66c8934117a","Type":"ContainerDied","Data":"b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d"} Nov 03 18:25:56.256285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:25:56.256148 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7add23362990aef7c1731c6b9b01dfdc0b4d817654a90d88d5e9059df98277d" Nov 03 18:26:02.674230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.674192 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5"] Nov 03 18:26:02.674778 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.674761 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6cf26149-7df2-4580-a396-d66c8934117a" containerName="revoke-keys" Nov 03 18:26:02.674833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.674780 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cf26149-7df2-4580-a396-d66c8934117a" containerName="revoke-keys" Nov 03 18:26:02.674869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.674851 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cf26149-7df2-4580-a396-d66c8934117a" containerName="revoke-keys" Nov 03 18:26:02.678003 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.677978 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.781328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.781291 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:26:02.781328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.781319 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:26:02.783631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.783609 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5"] Nov 03 18:26:02.820946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.820913 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.821134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.820974 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.821134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.821083 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmf5p\" (UniqueName: \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.922649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.922609 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.922865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.922679 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.922865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.922731 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmf5p\" (UniqueName: \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.923432 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.923411 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.925211 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.925157 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.986355 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.986321 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmf5p\" (UniqueName: \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\") pod \"maas-api-revoke-sub-d3fba056-4qnz5\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:02.993175 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:02.993151 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:03.370546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:03.370515 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5"] Nov 03 18:26:03.372837 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:26:03.372810 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cd441e2_4bc0_4404_9704_11fbd3714d0d.slice/crio-130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b WatchSource:0}: Error finding container 130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b: Status 404 returned error can't find the container with id 130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b Nov 03 18:26:04.302818 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:04.302782 4980 generic.go:356] "Generic (PLEG): container finished" podID="6cd441e2-4bc0-4404-9704-11fbd3714d0d" containerID="a9fc487f39f8d3473427086af4bb4a96714173c42c05a0a5b7a11b97c0185cb7" exitCode=0 Nov 03 18:26:04.303204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:04.302860 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" event={"ID":"6cd441e2-4bc0-4404-9704-11fbd3714d0d","Type":"ContainerDied","Data":"a9fc487f39f8d3473427086af4bb4a96714173c42c05a0a5b7a11b97c0185cb7"} Nov 03 18:26:04.303204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:04.302911 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" event={"ID":"6cd441e2-4bc0-4404-9704-11fbd3714d0d","Type":"ContainerStarted","Data":"130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b"} Nov 03 18:26:05.494589 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.494545 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:05.649103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.648987 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\") pod \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " Nov 03 18:26:05.649103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.649076 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\") pod \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " Nov 03 18:26:05.649369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.649207 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\" (UniqueName: \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\") pod \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\" (UID: \"6cd441e2-4bc0-4404-9704-11fbd3714d0d\") " Nov 03 18:26:05.649561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.649522 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca" pod "6cd441e2-4bc0-4404-9704-11fbd3714d0d" (UID: "6cd441e2-4bc0-4404-9704-11fbd3714d0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:26:05.652484 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.652447 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p" pod "6cd441e2-4bc0-4404-9704-11fbd3714d0d" (UID: "6cd441e2-4bc0-4404-9704-11fbd3714d0d"). InnerVolumeSpecName "kube-api-access-bmf5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:26:05.652768 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.652726 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca" pod "6cd441e2-4bc0-4404-9704-11fbd3714d0d" (UID: "6cd441e2-4bc0-4404-9704-11fbd3714d0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:26:05.751234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.751183 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:05.751234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.751237 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cd441e2-4bc0-4404-9704-11fbd3714d0d-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:05.751559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:05.751257 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmf5p\" (UniqueName: \"kubernetes.io/projected/6cd441e2-4bc0-4404-9704-11fbd3714d0d-kube-api-access-bmf5p\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:06.346596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:06.345816 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" Nov 03 18:26:06.346783 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:06.346751 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5" event={"ID":"6cd441e2-4bc0-4404-9704-11fbd3714d0d","Type":"ContainerDied","Data":"130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b"} Nov 03 18:26:06.346874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:06.346808 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="130c3d7df2de043c2347254248b8e53060e2d6cba9b789917dffe1f0ae0d597b" Nov 03 18:26:14.519716 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:26:14.519680 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:26:40.733232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.733189 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh"] Nov 03 18:26:40.733722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.733648 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6cd441e2-4bc0-4404-9704-11fbd3714d0d" containerName="revoke-keys" Nov 03 18:26:40.733722 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.733661 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cd441e2-4bc0-4404-9704-11fbd3714d0d" containerName="revoke-keys" Nov 03 18:26:40.733803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.733752 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cd441e2-4bc0-4404-9704-11fbd3714d0d" containerName="revoke-keys" Nov 03 18:26:40.736784 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.736767 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.781246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.781207 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.781246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.781254 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54jl2\" (UniqueName: \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.781502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.781418 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.840180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.840149 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:26:40.840399 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.840215 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:26:40.842705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.842677 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh"] Nov 03 18:26:40.881990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.881953 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.882180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.882046 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.882180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.882072 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54jl2\" (UniqueName: \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.882685 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.882651 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.884745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.884721 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:40.945625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:40.945560 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54jl2\" (UniqueName: \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\") pod \"maas-api-revoke-sub-ae594979-nmfnh\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:41.049894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:41.049789 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:41.429958 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:41.429933 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh"] Nov 03 18:26:41.432530 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:26:41.432499 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16f99b26_28b5_4536_b409_271a410b741a.slice/crio-67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850 WatchSource:0}: Error finding container 67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850: Status 404 returned error can't find the container with id 67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850 Nov 03 18:26:41.593048 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:41.593014 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" event={"ID":"16f99b26-28b5-4536-b409-271a410b741a","Type":"ContainerStarted","Data":"67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850"} Nov 03 18:26:42.600096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:42.600011 4980 generic.go:356] "Generic (PLEG): container finished" podID="16f99b26-28b5-4536-b409-271a410b741a" containerID="1fbc128b8cc42f21fffc6c0da06042d519c4335d20710f4c7782e22fbdde2eee" exitCode=0 Nov 03 18:26:42.600478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:42.600101 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" event={"ID":"16f99b26-28b5-4536-b409-271a410b741a","Type":"ContainerDied","Data":"1fbc128b8cc42f21fffc6c0da06042d519c4335d20710f4c7782e22fbdde2eee"} Nov 03 18:26:43.750251 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.750225 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:43.803908 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.803879 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\" (UniqueName: \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\") pod \"16f99b26-28b5-4536-b409-271a410b741a\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " Nov 03 18:26:43.804086 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.803948 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\") pod \"16f99b26-28b5-4536-b409-271a410b741a\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " Nov 03 18:26:43.804086 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.804026 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\") pod \"16f99b26-28b5-4536-b409-271a410b741a\" (UID: \"16f99b26-28b5-4536-b409-271a410b741a\") " Nov 03 18:26:43.804402 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.804378 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca" pod "16f99b26-28b5-4536-b409-271a410b741a" (UID: "16f99b26-28b5-4536-b409-271a410b741a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:26:43.806260 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.806230 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca" pod "16f99b26-28b5-4536-b409-271a410b741a" (UID: "16f99b26-28b5-4536-b409-271a410b741a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:26:43.806371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.806272 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2" pod "16f99b26-28b5-4536-b409-271a410b741a" (UID: "16f99b26-28b5-4536-b409-271a410b741a"). InnerVolumeSpecName "kube-api-access-54jl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:26:43.905079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.904991 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/16f99b26-28b5-4536-b409-271a410b741a-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:43.905079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.905027 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54jl2\" (UniqueName: \"kubernetes.io/projected/16f99b26-28b5-4536-b409-271a410b741a-kube-api-access-54jl2\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:43.905079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:43.905040 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/16f99b26-28b5-4536-b409-271a410b741a-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:26:44.520811 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:26:44.520770 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:26:44.612006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:44.611913 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" Nov 03 18:26:44.612006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:44.611949 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh" event={"ID":"16f99b26-28b5-4536-b409-271a410b741a","Type":"ContainerDied","Data":"67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850"} Nov 03 18:26:44.612212 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:26:44.612007 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67adf6bc80cc7674be1c88c85845b3bb5d584fb4e88eb04b19254f3a4231c850" Nov 03 18:27:12.258799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.258705 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn"] Nov 03 18:27:12.259344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.259286 4980 cpu_manager.go:414] "containerMap: removing container" podUID="16f99b26-28b5-4536-b409-271a410b741a" containerName="revoke-keys" Nov 03 18:27:12.259344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.259304 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16f99b26-28b5-4536-b409-271a410b741a" containerName="revoke-keys" Nov 03 18:27:12.259469 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.259422 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="16f99b26-28b5-4536-b409-271a410b741a" containerName="revoke-keys" Nov 03 18:27:12.262459 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.262441 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.354940 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.354899 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.355125 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.355081 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxhl6\" (UniqueName: \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.355186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.355141 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.365942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.365922 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:27:12.365942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.365930 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:27:12.368350 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.368326 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn"] Nov 03 18:27:12.455916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.455879 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.456098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.455975 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.456098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.456084 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxhl6\" (UniqueName: \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.456497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.456475 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.458466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.458446 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.519187 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.519115 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxhl6\" (UniqueName: \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\") pod \"maas-api-revoke-sub-a4aef7d7-dxvzn\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.578286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.578245 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:12.956158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:12.956134 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn"] Nov 03 18:27:12.958548 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:27:12.958520 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74d7b09_b818_4d64_b01f_2f97248336ca.slice/crio-c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b WatchSource:0}: Error finding container c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b: Status 404 returned error can't find the container with id c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b Nov 03 18:27:13.784701 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:13.784662 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" event={"ID":"a74d7b09-b818-4d64-b01f-2f97248336ca","Type":"ContainerStarted","Data":"b6c984c60185bd7975920a2030eaa42e82160dfa30dbe3acb7d4870285dfd60a"} Nov 03 18:27:13.785099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:13.784746 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" event={"ID":"a74d7b09-b818-4d64-b01f-2f97248336ca","Type":"ContainerStarted","Data":"c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b"} Nov 03 18:27:13.904548 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:13.904496 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" podStartSLOduration=1.904482234 podStartE2EDuration="1.904482234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:27:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:27:13.903524998 +0000 UTC m=+3213.732608601" watchObservedRunningTime="2026-11-03 18:27:13.904482234 +0000 UTC m=+3213.733565836" Nov 03 18:27:14.520055 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:27:14.520016 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:27:14.792264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:14.792179 4980 generic.go:356] "Generic (PLEG): container finished" podID="a74d7b09-b818-4d64-b01f-2f97248336ca" containerID="b6c984c60185bd7975920a2030eaa42e82160dfa30dbe3acb7d4870285dfd60a" exitCode=0 Nov 03 18:27:14.792689 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:14.792264 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" event={"ID":"a74d7b09-b818-4d64-b01f-2f97248336ca","Type":"ContainerDied","Data":"b6c984c60185bd7975920a2030eaa42e82160dfa30dbe3acb7d4870285dfd60a"} Nov 03 18:27:15.940762 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:15.940734 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:16.090006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.089923 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\") pod \"a74d7b09-b818-4d64-b01f-2f97248336ca\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " Nov 03 18:27:16.090006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.089971 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\") pod \"a74d7b09-b818-4d64-b01f-2f97248336ca\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " Nov 03 18:27:16.090278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.090074 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\" (UniqueName: \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\") pod \"a74d7b09-b818-4d64-b01f-2f97248336ca\" (UID: \"a74d7b09-b818-4d64-b01f-2f97248336ca\") " Nov 03 18:27:16.090373 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.090331 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca" pod "a74d7b09-b818-4d64-b01f-2f97248336ca" (UID: "a74d7b09-b818-4d64-b01f-2f97248336ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:27:16.092305 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.092278 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca" pod "a74d7b09-b818-4d64-b01f-2f97248336ca" (UID: "a74d7b09-b818-4d64-b01f-2f97248336ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:27:16.092424 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.092346 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6" pod "a74d7b09-b818-4d64-b01f-2f97248336ca" (UID: "a74d7b09-b818-4d64-b01f-2f97248336ca"). InnerVolumeSpecName "kube-api-access-mxhl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:27:16.191230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.191199 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:16.191230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.191228 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a74d7b09-b818-4d64-b01f-2f97248336ca-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:16.191430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.191238 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxhl6\" (UniqueName: \"kubernetes.io/projected/a74d7b09-b818-4d64-b01f-2f97248336ca-kube-api-access-mxhl6\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:16.803044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.803013 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" Nov 03 18:27:16.803253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.803049 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn" event={"ID":"a74d7b09-b818-4d64-b01f-2f97248336ca","Type":"ContainerDied","Data":"c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b"} Nov 03 18:27:16.803253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:16.803099 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c19915c389bf65295f231a522373a69b1fc5c2fd7dab651ac567d969434f087b" Nov 03 18:27:30.061616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:30.061582 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c"] Nov 03 18:27:30.102204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:30.102171 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79986443-hxp9c"] Nov 03 18:27:30.302636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:30.302596 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ecd753-1a1f-433d-9bbf-81fb884a4ce2" path="/var/lib/kubelet/pods/08ecd753-1a1f-433d-9bbf-81fb884a4ce2/volumes" Nov 03 18:27:44.520693 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:27:44.520657 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:27:46.273341 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.273283 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz"] Nov 03 18:27:46.273799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.273770 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a74d7b09-b818-4d64-b01f-2f97248336ca" containerName="revoke-keys" Nov 03 18:27:46.273799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.273781 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74d7b09-b818-4d64-b01f-2f97248336ca" containerName="revoke-keys" Nov 03 18:27:46.273886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.273838 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74d7b09-b818-4d64-b01f-2f97248336ca" containerName="revoke-keys" Nov 03 18:27:46.277113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.277097 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.380181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.380147 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:27:46.380352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.380185 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:27:46.382644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.382622 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz"] Nov 03 18:27:46.408861 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.408834 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.409016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.408968 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.409076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.409068 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5q9v\" (UniqueName: \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.509735 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.509704 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.509904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.509768 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5q9v\" (UniqueName: \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.509904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.509786 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.510414 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.510389 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.512325 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.512304 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.573129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.573075 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5q9v\" (UniqueName: \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\") pod \"maas-api-revoke-sub-79805ca9-g2rjz\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.590865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.590841 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:46.970028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.970004 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz"] Nov 03 18:27:46.971945 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:27:46.971917 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d14d393_d8bd_458e_83d3_155c080ac688.slice/crio-8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c WatchSource:0}: Error finding container 8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c: Status 404 returned error can't find the container with id 8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c Nov 03 18:27:46.980379 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:46.980355 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" event={"ID":"5d14d393-d8bd-458e-83d3-155c080ac688","Type":"ContainerStarted","Data":"8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c"} Nov 03 18:27:47.986776 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:47.986743 4980 generic.go:356] "Generic (PLEG): container finished" podID="5d14d393-d8bd-458e-83d3-155c080ac688" containerID="3ea1fcc23c18693da68d43a030a17ef70d6eaaa31bd64d05f7b717d7dd1e2bf8" exitCode=0 Nov 03 18:27:47.987185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:47.986832 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" event={"ID":"5d14d393-d8bd-458e-83d3-155c080ac688","Type":"ContainerDied","Data":"3ea1fcc23c18693da68d43a030a17ef70d6eaaa31bd64d05f7b717d7dd1e2bf8"} Nov 03 18:27:49.136147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.136120 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:49.236769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.236738 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\") pod \"5d14d393-d8bd-458e-83d3-155c080ac688\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " Nov 03 18:27:49.236919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.236850 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\" (UniqueName: \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\") pod \"5d14d393-d8bd-458e-83d3-155c080ac688\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " Nov 03 18:27:49.236919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.236892 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\") pod \"5d14d393-d8bd-458e-83d3-155c080ac688\" (UID: \"5d14d393-d8bd-458e-83d3-155c080ac688\") " Nov 03 18:27:49.237083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.237059 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca" pod "5d14d393-d8bd-458e-83d3-155c080ac688" (UID: "5d14d393-d8bd-458e-83d3-155c080ac688"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:27:49.237206 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.237190 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:49.239162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.239135 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca" pod "5d14d393-d8bd-458e-83d3-155c080ac688" (UID: "5d14d393-d8bd-458e-83d3-155c080ac688"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:27:49.239270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.239191 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v" pod "5d14d393-d8bd-458e-83d3-155c080ac688" (UID: "5d14d393-d8bd-458e-83d3-155c080ac688"). InnerVolumeSpecName "kube-api-access-z5q9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:27:49.338502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.338469 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5q9v\" (UniqueName: \"kubernetes.io/projected/5d14d393-d8bd-458e-83d3-155c080ac688-kube-api-access-z5q9v\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:49.338502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.338498 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d14d393-d8bd-458e-83d3-155c080ac688-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:27:49.999020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.998992 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" Nov 03 18:27:49.999182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.999034 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz" event={"ID":"5d14d393-d8bd-458e-83d3-155c080ac688","Type":"ContainerDied","Data":"8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c"} Nov 03 18:27:49.999182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:27:49.999084 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8187d920366a17cac1a723ffae2a3a35eeff0ce4f0ee35c133eefc0d3921904c" Nov 03 18:28:10.077588 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.077533 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q"] Nov 03 18:28:10.078351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.078074 4980 cpu_manager.go:414] "containerMap: removing container" podUID="5d14d393-d8bd-458e-83d3-155c080ac688" containerName="revoke-keys" Nov 03 18:28:10.078351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.078089 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d14d393-d8bd-458e-83d3-155c080ac688" containerName="revoke-keys" Nov 03 18:28:10.078351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.078176 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d14d393-d8bd-458e-83d3-155c080ac688" containerName="revoke-keys" Nov 03 18:28:10.081896 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.081873 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.112815 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.112788 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.112973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.112823 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.112973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.112846 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mb8f\" (UniqueName: \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.161566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.161535 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:28:10.161736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.161534 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:28:10.187723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.187692 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q"] Nov 03 18:28:10.213857 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.213822 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.213996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.213864 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.213996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.213887 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mb8f\" (UniqueName: \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.214667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.214497 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.216278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.216256 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.277220 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.277193 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mb8f\" (UniqueName: \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\") pod \"maas-api-revoke-sub-0b7e8edb-n9c8q\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.397609 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.397518 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:10.782081 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:10.782047 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q"] Nov 03 18:28:10.783798 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:28:10.783766 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod068041b2_1944_4608_a9ff_2341a089b6f5.slice/crio-ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8 WatchSource:0}: Error finding container ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8: Status 404 returned error can't find the container with id ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8 Nov 03 18:28:11.161226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:11.161190 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" event={"ID":"068041b2-1944-4608-a9ff-2341a089b6f5","Type":"ContainerStarted","Data":"9f44674bacafad9ff913c1053ae6f9df3a949c1f761060cc19acf0b95b338de4"} Nov 03 18:28:11.161658 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:11.161240 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" event={"ID":"068041b2-1944-4608-a9ff-2341a089b6f5","Type":"ContainerStarted","Data":"ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8"} Nov 03 18:28:11.280774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:11.280719 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" podStartSLOduration=1.280705138 podStartE2EDuration="1.280705138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:11.280022523 +0000 UTC m=+3271.109106127" watchObservedRunningTime="2026-11-03 18:28:11.280705138 +0000 UTC m=+3271.109788740" Nov 03 18:28:12.167549 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:12.167508 4980 generic.go:356] "Generic (PLEG): container finished" podID="068041b2-1944-4608-a9ff-2341a089b6f5" containerID="9f44674bacafad9ff913c1053ae6f9df3a949c1f761060cc19acf0b95b338de4" exitCode=0 Nov 03 18:28:12.167982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:12.167594 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" event={"ID":"068041b2-1944-4608-a9ff-2341a089b6f5","Type":"ContainerDied","Data":"9f44674bacafad9ff913c1053ae6f9df3a949c1f761060cc19acf0b95b338de4"} Nov 03 18:28:13.317619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.317593 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:13.346843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.346815 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\") pod \"068041b2-1944-4608-a9ff-2341a089b6f5\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " Nov 03 18:28:13.346973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.346950 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\" (UniqueName: \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\") pod \"068041b2-1944-4608-a9ff-2341a089b6f5\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " Nov 03 18:28:13.347039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.347001 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\") pod \"068041b2-1944-4608-a9ff-2341a089b6f5\" (UID: \"068041b2-1944-4608-a9ff-2341a089b6f5\") " Nov 03 18:28:13.347352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.347322 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca" pod "068041b2-1944-4608-a9ff-2341a089b6f5" (UID: "068041b2-1944-4608-a9ff-2341a089b6f5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:28:13.348953 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.348931 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca" pod "068041b2-1944-4608-a9ff-2341a089b6f5" (UID: "068041b2-1944-4608-a9ff-2341a089b6f5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:28:13.349163 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.349144 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f" pod "068041b2-1944-4608-a9ff-2341a089b6f5" (UID: "068041b2-1944-4608-a9ff-2341a089b6f5"). InnerVolumeSpecName "kube-api-access-5mb8f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:28:13.448107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.448067 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:13.448107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.448095 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mb8f\" (UniqueName: \"kubernetes.io/projected/068041b2-1944-4608-a9ff-2341a089b6f5-kube-api-access-5mb8f\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:13.448107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:13.448106 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/068041b2-1944-4608-a9ff-2341a089b6f5-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:14.179080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:14.179043 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" Nov 03 18:28:14.179080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:14.179069 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q" event={"ID":"068041b2-1944-4608-a9ff-2341a089b6f5","Type":"ContainerDied","Data":"ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8"} Nov 03 18:28:14.179293 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:14.179112 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed5f0a09469a457b39a056e0e6f3782bce1c6caa30a56ce0d70453b24f0300a8" Nov 03 18:28:14.519524 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:28:14.519481 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:28:21.486134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.486054 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k"] Nov 03 18:28:21.486967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.486946 4980 cpu_manager.go:414] "containerMap: removing container" podUID="068041b2-1944-4608-a9ff-2341a089b6f5" containerName="revoke-keys" Nov 03 18:28:21.487042 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.486969 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="068041b2-1944-4608-a9ff-2341a089b6f5" containerName="revoke-keys" Nov 03 18:28:21.487096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.487086 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="068041b2-1944-4608-a9ff-2341a089b6f5" containerName="revoke-keys" Nov 03 18:28:21.490608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.490590 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.545227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.545198 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:28:21.593965 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.593930 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:28:21.596785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.596762 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k"] Nov 03 18:28:21.628523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.628487 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.628769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.628603 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.628769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.628664 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wf56\" (UniqueName: \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.729726 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.729685 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.729919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.729787 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.729919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.729813 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wf56\" (UniqueName: \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.730555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.730486 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.732446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.732424 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.792973 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.792894 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wf56\" (UniqueName: \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\") pod \"maas-api-revoke-sub-3d4156f1-fpf9k\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:21.803708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:21.803685 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:22.184215 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:22.184186 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k"] Nov 03 18:28:22.186356 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:28:22.186313 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c925c02_d08d_4b53_9b8a_f24ebcdda65f.slice/crio-1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525 WatchSource:0}: Error finding container 1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525: Status 404 returned error can't find the container with id 1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525 Nov 03 18:28:22.221366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:22.221334 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" event={"ID":"0c925c02-d08d-4b53-9b8a-f24ebcdda65f","Type":"ContainerStarted","Data":"1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525"} Nov 03 18:28:23.228710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:23.228677 4980 generic.go:356] "Generic (PLEG): container finished" podID="0c925c02-d08d-4b53-9b8a-f24ebcdda65f" containerID="1958ac71c90feddaacab1bae165d18c7692eda2fe3cbb0c2f641561a26520432" exitCode=0 Nov 03 18:28:23.229136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:23.228777 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" event={"ID":"0c925c02-d08d-4b53-9b8a-f24ebcdda65f","Type":"ContainerDied","Data":"1958ac71c90feddaacab1bae165d18c7692eda2fe3cbb0c2f641561a26520432"} Nov 03 18:28:24.086675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:24.086627 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25"] Nov 03 18:28:24.101594 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:24.099142 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-rdf25"] Nov 03 18:28:24.322332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:24.321915 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b" path="/var/lib/kubelet/pods/9c8382d9-4fb4-40a3-aeff-6bbf1651cc9b/volumes" Nov 03 18:28:25.029880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.029834 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:25.171660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.169033 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\") pod \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " Nov 03 18:28:25.171660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.169137 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\" (UniqueName: \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\") pod \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " Nov 03 18:28:25.171660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.169232 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\") pod \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\" (UID: \"0c925c02-d08d-4b53-9b8a-f24ebcdda65f\") " Nov 03 18:28:25.216758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.216701 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca" pod "0c925c02-d08d-4b53-9b8a-f24ebcdda65f" (UID: "0c925c02-d08d-4b53-9b8a-f24ebcdda65f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:28:25.243286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.242329 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56" pod "0c925c02-d08d-4b53-9b8a-f24ebcdda65f" (UID: "0c925c02-d08d-4b53-9b8a-f24ebcdda65f"). InnerVolumeSpecName "kube-api-access-8wf56". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:28:25.243286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.243233 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca" pod "0c925c02-d08d-4b53-9b8a-f24ebcdda65f" (UID: "0c925c02-d08d-4b53-9b8a-f24ebcdda65f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:28:25.349598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.345331 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:25.349598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.345366 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wf56\" (UniqueName: \"kubernetes.io/projected/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-kube-api-access-8wf56\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:25.349598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.345384 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0c925c02-d08d-4b53-9b8a-f24ebcdda65f-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:25.441651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.440897 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" event={"ID":"0c925c02-d08d-4b53-9b8a-f24ebcdda65f","Type":"ContainerDied","Data":"1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525"} Nov 03 18:28:25.441651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.440959 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c1ee1a8895ca7d0a2fefe46d63cb1ee0f5435f5a40874cbdb7003ec45e7b525" Nov 03 18:28:25.441651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:25.441034 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k" Nov 03 18:28:41.172884 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:41.172858 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:28:41.177709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:41.177686 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:28:41.235532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:41.235504 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:28:41.240871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:41.240853 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:28:42.624431 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:42.624398 4980 scope.go:122] "RemoveContainer" containerID="19ed05017a0229530ce876ba257f11079501c58a7645dcb97d343275b5060fc5" Nov 03 18:28:44.639029 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:28:44.638986 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:28:45.729723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:45.729688 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-bftjk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:28:45.730177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:45.729759 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-bftjk" podUID="32c6c7af-e170-4045-bb87-4cca726a11f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.171:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:28:52.166907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.166874 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k"] Nov 03 18:28:52.217181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.217146 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-fpf9k"] Nov 03 18:28:52.293679 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.293649 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c925c02-d08d-4b53-9b8a-f24ebcdda65f" path="/var/lib/kubelet/pods/0c925c02-d08d-4b53-9b8a-f24ebcdda65f/volumes" Nov 03 18:28:52.293951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.293938 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq"] Nov 03 18:28:52.294332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.294321 4980 cpu_manager.go:414] "containerMap: removing container" podUID="0c925c02-d08d-4b53-9b8a-f24ebcdda65f" containerName="revoke-keys" Nov 03 18:28:52.294378 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.294350 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c925c02-d08d-4b53-9b8a-f24ebcdda65f" containerName="revoke-keys" Nov 03 18:28:52.294461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.294451 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c925c02-d08d-4b53-9b8a-f24ebcdda65f" containerName="revoke-keys" Nov 03 18:28:52.298864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.298841 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.353589 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.353547 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:28:52.353724 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.353547 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:28:52.356068 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.356038 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq"] Nov 03 18:28:52.357044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.357019 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.357166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.357056 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.357166 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.357147 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bx4sw\" (UniqueName: \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.458492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.458454 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.458712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.458515 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.458712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.458550 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bx4sw\" (UniqueName: \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.459216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.459191 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.461158 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.461140 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.521720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.521689 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bx4sw\" (UniqueName: \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\") pod \"maas-api-revoke-sub-3d4156f1-jdjhq\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.612758 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.612722 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:52.990028 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:52.989998 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq"] Nov 03 18:28:52.992142 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:28:52.992099 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3228df80_081d_4b73_9692_497c4ba49215.slice/crio-60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5 WatchSource:0}: Error finding container 60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5: Status 404 returned error can't find the container with id 60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5 Nov 03 18:28:53.718032 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:53.717999 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" event={"ID":"3228df80-081d-4b73-9692-497c4ba49215","Type":"ContainerStarted","Data":"30f19b95feb0a68274179fe6515b32280aa051e8fe8439c8d16dae31cdbf3df9"} Nov 03 18:28:53.718493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:53.718053 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" event={"ID":"3228df80-081d-4b73-9692-497c4ba49215","Type":"ContainerStarted","Data":"60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5"} Nov 03 18:28:53.837333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:53.837284 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" podStartSLOduration=1.837269375 podStartE2EDuration="1.837269375s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:28:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:28:53.836093214 +0000 UTC m=+3313.665176817" watchObservedRunningTime="2026-11-03 18:28:53.837269375 +0000 UTC m=+3313.666352977" Nov 03 18:28:54.724318 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:54.724278 4980 generic.go:356] "Generic (PLEG): container finished" podID="3228df80-081d-4b73-9692-497c4ba49215" containerID="30f19b95feb0a68274179fe6515b32280aa051e8fe8439c8d16dae31cdbf3df9" exitCode=0 Nov 03 18:28:54.724761 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:54.724361 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" event={"ID":"3228df80-081d-4b73-9692-497c4ba49215","Type":"ContainerDied","Data":"30f19b95feb0a68274179fe6515b32280aa051e8fe8439c8d16dae31cdbf3df9"} Nov 03 18:28:55.859014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.858987 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:55.992481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.992405 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\") pod \"3228df80-081d-4b73-9692-497c4ba49215\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " Nov 03 18:28:55.992481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.992443 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\" (UniqueName: \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\") pod \"3228df80-081d-4b73-9692-497c4ba49215\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " Nov 03 18:28:55.992741 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.992525 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\") pod \"3228df80-081d-4b73-9692-497c4ba49215\" (UID: \"3228df80-081d-4b73-9692-497c4ba49215\") " Nov 03 18:28:55.992920 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.992894 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca" pod "3228df80-081d-4b73-9692-497c4ba49215" (UID: "3228df80-081d-4b73-9692-497c4ba49215"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:28:55.994775 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.994751 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca" pod "3228df80-081d-4b73-9692-497c4ba49215" (UID: "3228df80-081d-4b73-9692-497c4ba49215"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:28:55.994875 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:55.994833 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw" pod "3228df80-081d-4b73-9692-497c4ba49215" (UID: "3228df80-081d-4b73-9692-497c4ba49215"). InnerVolumeSpecName "kube-api-access-bx4sw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:28:56.094015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.093988 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3228df80-081d-4b73-9692-497c4ba49215-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:56.094142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.094017 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bx4sw\" (UniqueName: \"kubernetes.io/projected/3228df80-081d-4b73-9692-497c4ba49215-kube-api-access-bx4sw\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:56.094142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.094033 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3228df80-081d-4b73-9692-497c4ba49215-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:28:56.740087 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.740053 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" Nov 03 18:28:56.740278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.740058 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq" event={"ID":"3228df80-081d-4b73-9692-497c4ba49215","Type":"ContainerDied","Data":"60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5"} Nov 03 18:28:56.740278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:28:56.740173 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60fda1043e79f95c5283326b084719a3a974addbb7ab2bc3ea47ead6645725d5" Nov 03 18:29:14.520021 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:29:14.519981 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:29:36.063598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:36.063035 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7"] Nov 03 18:29:36.119177 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:36.119142 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-ldrx7"] Nov 03 18:29:36.397921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:36.397837 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9963d6d8-3f02-4f7a-a3d0-ce226789db5d" path="/var/lib/kubelet/pods/9963d6d8-3f02-4f7a-a3d0-ce226789db5d/volumes" Nov 03 18:29:42.646493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.646461 4980 scope.go:122] "RemoveContainer" containerID="4de0bd4100b349eaa41de815b5743116186614c38e25d6c72fb28d87f968256a" Nov 03 18:29:42.793050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.793018 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz"] Nov 03 18:29:42.793444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.793431 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3228df80-081d-4b73-9692-497c4ba49215" containerName="revoke-keys" Nov 03 18:29:42.793500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.793445 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3228df80-081d-4b73-9692-497c4ba49215" containerName="revoke-keys" Nov 03 18:29:42.793546 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.793538 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3228df80-081d-4b73-9692-497c4ba49215" containerName="revoke-keys" Nov 03 18:29:42.797990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.797972 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:42.857646 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.857619 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz"] Nov 03 18:29:42.904161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.904083 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:29:42.904161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.904097 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:29:42.909972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.909944 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvhp\" (UniqueName: \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:42.910093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.909987 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:42.910093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:42.910018 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.010879 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.010851 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvhp\" (UniqueName: \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.011027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.010890 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.011027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.010920 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.011518 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.011488 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.013398 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.013380 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.073770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.073747 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvhp\" (UniqueName: \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\") pod \"maas-api-revoke-sub-e2fc9c5b-cw5nz\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.111317 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.111288 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:43.487631 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:43.487602 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz"] Nov 03 18:29:43.489754 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:29:43.489729 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64f171bb_d7e6_4e0a_8ae3_77b4c44c2386.slice/crio-daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40 WatchSource:0}: Error finding container daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40: Status 404 returned error can't find the container with id daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40 Nov 03 18:29:44.006694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:44.006660 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" event={"ID":"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386","Type":"ContainerStarted","Data":"5a9ec6b523cedb91a8bcecea20a1271399a267163720b574ce67b2572920046e"} Nov 03 18:29:44.007101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:44.006711 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" event={"ID":"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386","Type":"ContainerStarted","Data":"daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40"} Nov 03 18:29:44.126955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:44.126849 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" podStartSLOduration=2.126835007 podStartE2EDuration="2.126835007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:44.1241501 +0000 UTC m=+3363.953233701" watchObservedRunningTime="2026-11-03 18:29:44.126835007 +0000 UTC m=+3363.955918629" Nov 03 18:29:44.520025 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:29:44.519992 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:29:45.013667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:45.013635 4980 generic.go:356] "Generic (PLEG): container finished" podID="64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" containerID="5a9ec6b523cedb91a8bcecea20a1271399a267163720b574ce67b2572920046e" exitCode=0 Nov 03 18:29:45.014050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:45.013685 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" event={"ID":"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386","Type":"ContainerDied","Data":"5a9ec6b523cedb91a8bcecea20a1271399a267163720b574ce67b2572920046e"} Nov 03 18:29:46.165140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.165113 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:46.341563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.341490 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\") pod \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " Nov 03 18:29:46.341728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.341603 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\") pod \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " Nov 03 18:29:46.341728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.341637 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\" (UniqueName: \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\") pod \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\" (UID: \"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386\") " Nov 03 18:29:46.341919 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.341895 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca" pod "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" (UID: "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:29:46.341989 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.341975 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:46.343769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.343742 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca" pod "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" (UID: "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:29:46.343874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.343822 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp" pod "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" (UID: "64f171bb-d7e6-4e0a-8ae3-77b4c44c2386"). InnerVolumeSpecName "kube-api-access-ccvhp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:29:46.443390 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.443357 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:46.443390 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:46.443388 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvhp\" (UniqueName: \"kubernetes.io/projected/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386-kube-api-access-ccvhp\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:47.025059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:47.025030 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" Nov 03 18:29:47.025059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:47.025048 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz" event={"ID":"64f171bb-d7e6-4e0a-8ae3-77b4c44c2386","Type":"ContainerDied","Data":"daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40"} Nov 03 18:29:47.025283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:47.025094 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="daca9d877663f37526441a6617b81de79689d7ee3b90a03d49b5ed5408374a40" Nov 03 18:29:53.573733 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.573649 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p"] Nov 03 18:29:53.574127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.574079 4980 cpu_manager.go:414] "containerMap: removing container" podUID="64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" containerName="revoke-keys" Nov 03 18:29:53.574127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.574092 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" containerName="revoke-keys" Nov 03 18:29:53.574223 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.574170 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" containerName="revoke-keys" Nov 03 18:29:53.577427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.577399 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.632445 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.632414 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:29:53.681064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.681033 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:29:53.683707 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.683682 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p"] Nov 03 18:29:53.714502 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.714474 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.714644 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.714544 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.714690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.714670 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdbwk\" (UniqueName: \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.815805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.815773 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.815980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.815858 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.815980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.815913 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdbwk\" (UniqueName: \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.816463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.816437 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.818268 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.818250 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.879293 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.879219 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdbwk\" (UniqueName: \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\") pod \"maas-api-revoke-sub-5a29efb3-72b5p\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:53.892089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:53.892066 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:54.273745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:54.273715 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p"] Nov 03 18:29:54.275913 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:29:54.275884 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69923fc6_93de_448f_8abf_592d3666e9d1.slice/crio-ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa WatchSource:0}: Error finding container ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa: Status 404 returned error can't find the container with id ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa Nov 03 18:29:55.070542 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:55.070501 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" event={"ID":"69923fc6-93de-448f-8abf-592d3666e9d1","Type":"ContainerStarted","Data":"8b85d57f9b131a629ba7a616630d85463669ece3a0a8068d9f0285401d9ea593"} Nov 03 18:29:55.070982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:55.070564 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" event={"ID":"69923fc6-93de-448f-8abf-592d3666e9d1","Type":"ContainerStarted","Data":"ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa"} Nov 03 18:29:55.191633 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:55.191560 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" podStartSLOduration=2.191545998 podStartE2EDuration="2.191545998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:29:55.190010736 +0000 UTC m=+3375.019094337" watchObservedRunningTime="2026-11-03 18:29:55.191545998 +0000 UTC m=+3375.020629600" Nov 03 18:29:56.078022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:56.077986 4980 generic.go:356] "Generic (PLEG): container finished" podID="69923fc6-93de-448f-8abf-592d3666e9d1" containerID="8b85d57f9b131a629ba7a616630d85463669ece3a0a8068d9f0285401d9ea593" exitCode=0 Nov 03 18:29:56.078494 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:56.078090 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" event={"ID":"69923fc6-93de-448f-8abf-592d3666e9d1","Type":"ContainerDied","Data":"8b85d57f9b131a629ba7a616630d85463669ece3a0a8068d9f0285401d9ea593"} Nov 03 18:29:57.912921 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:57.912889 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:58.044914 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.044883 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\") pod \"69923fc6-93de-448f-8abf-592d3666e9d1\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " Nov 03 18:29:58.045298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.045272 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\" (UniqueName: \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\") pod \"69923fc6-93de-448f-8abf-592d3666e9d1\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " Nov 03 18:29:58.045496 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.045476 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\") pod \"69923fc6-93de-448f-8abf-592d3666e9d1\" (UID: \"69923fc6-93de-448f-8abf-592d3666e9d1\") " Nov 03 18:29:58.046707 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.046674 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca" pod "69923fc6-93de-448f-8abf-592d3666e9d1" (UID: "69923fc6-93de-448f-8abf-592d3666e9d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:29:58.167611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.167544 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:58.273597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.267126 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" event={"ID":"69923fc6-93de-448f-8abf-592d3666e9d1","Type":"ContainerDied","Data":"ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa"} Nov 03 18:29:58.273597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.267189 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba0949c0ec447dd31296f1dd8323a9a83f42e9285cd2e972014757c9fd5573fa" Nov 03 18:29:58.273597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.267269 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p" Nov 03 18:29:58.280871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.280835 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk" pod "69923fc6-93de-448f-8abf-592d3666e9d1" (UID: "69923fc6-93de-448f-8abf-592d3666e9d1"). InnerVolumeSpecName "kube-api-access-fdbwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:29:58.316520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.315816 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca" pod "69923fc6-93de-448f-8abf-592d3666e9d1" (UID: "69923fc6-93de-448f-8abf-592d3666e9d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:29:58.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.399412 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdbwk\" (UniqueName: \"kubernetes.io/projected/69923fc6-93de-448f-8abf-592d3666e9d1-kube-api-access-fdbwk\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:58.399649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.399448 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/69923fc6-93de-448f-8abf-592d3666e9d1-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:29:58.730597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.729664 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:29:58.730597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.729726 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:29:58.749703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.749664 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:29:58.749938 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:29:58.749725 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:30:00.165747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.165709 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:00.166178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.166155 4980 cpu_manager.go:414] "containerMap: removing container" podUID="69923fc6-93de-448f-8abf-592d3666e9d1" containerName="revoke-keys" Nov 03 18:30:00.166178 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.166167 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69923fc6-93de-448f-8abf-592d3666e9d1" containerName="revoke-keys" Nov 03 18:30:00.166279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.166242 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="69923fc6-93de-448f-8abf-592d3666e9d1" containerName="revoke-keys" Nov 03 18:30:00.169646 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.169624 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.224806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.224778 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:30:00.238129 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.238075 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.238517 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.238485 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7mhg\" (UniqueName: \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.277094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.277060 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:00.340199 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.340168 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7mhg\" (UniqueName: \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.340391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.340260 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.340942 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.340921 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.404475 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.404437 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7mhg\" (UniqueName: \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\") pod \"maas-api-key-cleanup-29895510-qdqw6\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.485958 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.485934 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:00.865471 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:00.865433 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:00.868248 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:30:00.868219 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb758195_1b8e_4b07_81a0_a40406583712.slice/crio-97da1dcb8e0cd865cfcc65fb77475d5037a1ddadcea07a84c20d43d6f5dedb04 WatchSource:0}: Error finding container 97da1dcb8e0cd865cfcc65fb77475d5037a1ddadcea07a84c20d43d6f5dedb04: Status 404 returned error can't find the container with id 97da1dcb8e0cd865cfcc65fb77475d5037a1ddadcea07a84c20d43d6f5dedb04 Nov 03 18:30:01.285983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:01.285938 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerStarted","Data":"7b591936fdb7ea46206833ab16256a63ac8442e77e0e4e6b96dce6c900014a4e"} Nov 03 18:30:01.286404 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:01.285999 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerStarted","Data":"97da1dcb8e0cd865cfcc65fb77475d5037a1ddadcea07a84c20d43d6f5dedb04"} Nov 03 18:30:01.405386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:01.405325 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" podStartSLOduration=1.405312053 podStartE2EDuration="1.405312053s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:01.404156501 +0000 UTC m=+3381.233240105" watchObservedRunningTime="2026-11-03 18:30:01.405312053 +0000 UTC m=+3381.234395656" Nov 03 18:30:02.292859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:02.292824 4980 generic.go:356] "Generic (PLEG): container finished" podID="fb758195-1b8e-4b07-81a0-a40406583712" containerID="7b591936fdb7ea46206833ab16256a63ac8442e77e0e4e6b96dce6c900014a4e" exitCode=60 Nov 03 18:30:02.293280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:02.292876 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerDied","Data":"7b591936fdb7ea46206833ab16256a63ac8442e77e0e4e6b96dce6c900014a4e"} Nov 03 18:30:02.293280 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:02.293074 4980 scope.go:122] "RemoveContainer" containerID="7b591936fdb7ea46206833ab16256a63ac8442e77e0e4e6b96dce6c900014a4e" Nov 03 18:30:03.299959 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:03.299924 4980 generic.go:356] "Generic (PLEG): container finished" podID="fb758195-1b8e-4b07-81a0-a40406583712" containerID="97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a" exitCode=60 Nov 03 18:30:03.300359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:03.300006 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerDied","Data":"97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a"} Nov 03 18:30:03.300359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:03.300065 4980 scope.go:122] "RemoveContainer" containerID="7b591936fdb7ea46206833ab16256a63ac8442e77e0e4e6b96dce6c900014a4e" Nov 03 18:30:03.300359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:03.300287 4980 scope.go:122] "RemoveContainer" containerID="97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a" Nov 03 18:30:03.300532 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:30:03.300511 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-qdqw6_odh-ai-gateway-infra(fb758195-1b8e-4b07-81a0-a40406583712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" podUID="fb758195-1b8e-4b07-81a0-a40406583712" Nov 03 18:30:04.306939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:04.306912 4980 scope.go:122] "RemoveContainer" containerID="97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a" Nov 03 18:30:04.307398 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:30:04.307113 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-qdqw6_odh-ai-gateway-infra(fb758195-1b8e-4b07-81a0-a40406583712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" podUID="fb758195-1b8e-4b07-81a0-a40406583712" Nov 03 18:30:08.060769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:08.060733 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr"] Nov 03 18:30:08.110692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:08.110659 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-4pbkr"] Nov 03 18:30:08.292498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:08.292470 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c41dc66b-b16b-4ba9-bebe-dc191d06c746" path="/var/lib/kubelet/pods/c41dc66b-b16b-4ba9-bebe-dc191d06c746/volumes" Nov 03 18:30:10.651324 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:10.651286 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq"] Nov 03 18:30:10.701359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:10.701326 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-jdjhq"] Nov 03 18:30:12.293194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:12.293158 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3228df80-081d-4b73-9692-497c4ba49215" path="/var/lib/kubelet/pods/3228df80-081d-4b73-9692-497c4ba49215/volumes" Nov 03 18:30:14.519611 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:30:14.519558 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:30:16.287144 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:16.287109 4980 scope.go:122] "RemoveContainer" containerID="97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a" Nov 03 18:30:17.375303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:17.375227 4980 generic.go:356] "Generic (PLEG): container finished" podID="fb758195-1b8e-4b07-81a0-a40406583712" containerID="9697549876fc2167941cead4bf4411873e63f4cf80453e2be8c7605369397bb2" exitCode=60 Nov 03 18:30:17.375303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:17.375283 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerDied","Data":"9697549876fc2167941cead4bf4411873e63f4cf80453e2be8c7605369397bb2"} Nov 03 18:30:17.375886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:17.375332 4980 scope.go:122] "RemoveContainer" containerID="97bb2a635023f02ca369a8a2d1082d32cf8322cec3ca5426f9d2e1cc1fe7b88a" Nov 03 18:30:17.375886 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:17.375691 4980 scope.go:122] "RemoveContainer" containerID="9697549876fc2167941cead4bf4411873e63f4cf80453e2be8c7605369397bb2" Nov 03 18:30:17.375983 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:30:17.375950 4980 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29895510-qdqw6_odh-ai-gateway-infra(fb758195-1b8e-4b07-81a0-a40406583712)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" podUID="fb758195-1b8e-4b07-81a0-a40406583712" Nov 03 18:30:18.506499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.506462 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:18.650070 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.650045 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:18.732058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.732021 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\") pod \"fb758195-1b8e-4b07-81a0-a40406583712\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " Nov 03 18:30:18.732058 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.732059 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\" (UniqueName: \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\") pod \"fb758195-1b8e-4b07-81a0-a40406583712\" (UID: \"fb758195-1b8e-4b07-81a0-a40406583712\") " Nov 03 18:30:18.732453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.732430 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca" pod "fb758195-1b8e-4b07-81a0-a40406583712" (UID: "fb758195-1b8e-4b07-81a0-a40406583712"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:30:18.732563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.732547 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb758195-1b8e-4b07-81a0-a40406583712-maas-api-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:30:18.734364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.734326 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg" pod "fb758195-1b8e-4b07-81a0-a40406583712" (UID: "fb758195-1b8e-4b07-81a0-a40406583712"). InnerVolumeSpecName "kube-api-access-j7mhg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:30:18.833160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:18.833054 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7mhg\" (UniqueName: \"kubernetes.io/projected/fb758195-1b8e-4b07-81a0-a40406583712-kube-api-access-j7mhg\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:30:19.389038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:19.389004 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" Nov 03 18:30:19.389222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:19.389043 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6" event={"ID":"fb758195-1b8e-4b07-81a0-a40406583712","Type":"ContainerDied","Data":"97da1dcb8e0cd865cfcc65fb77475d5037a1ddadcea07a84c20d43d6f5dedb04"} Nov 03 18:30:19.389222 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:19.389107 4980 scope.go:122] "RemoveContainer" containerID="9697549876fc2167941cead4bf4411873e63f4cf80453e2be8c7605369397bb2" Nov 03 18:30:19.557616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:19.557586 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:19.562893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:19.562865 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29895510-qdqw6"] Nov 03 18:30:20.058448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:20.058411 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx"] Nov 03 18:30:20.061413 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:20.061387 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-7vhnx"] Nov 03 18:30:20.292669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:20.292640 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2680bc60-5cbb-4ebf-9388-574ee769124b" path="/var/lib/kubelet/pods/2680bc60-5cbb-4ebf-9388-574ee769124b/volumes" Nov 03 18:30:20.293035 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:20.293022 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb758195-1b8e-4b07-81a0-a40406583712" path="/var/lib/kubelet/pods/fb758195-1b8e-4b07-81a0-a40406583712/volumes" Nov 03 18:30:42.668326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:42.668291 4980 scope.go:122] "RemoveContainer" containerID="3fd0c53b38a1e76f9eade44a3ce9d605455114b014682da7c196a8ffd6cafb2b" Nov 03 18:30:43.406143 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406104 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:30:43.406535 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406521 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406619 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406536 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406623 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406630 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406637 4980 cpu_manager.go:414] "containerMap: removing container" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406643 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406766 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.406907 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.406778 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:43.411358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.411339 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.468991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.468949 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ca6dbf-openshift-default-dockercfg-54v45\"" Nov 03 18:30:43.520061 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.520025 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:30:43.522298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522272 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522309 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522430 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522357 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522492 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9jkk\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522538 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522668 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522653 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522730 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522684 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522730 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522720 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.522847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.522764 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624196 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624527 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624499 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624614 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624654 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624707 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624761 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9jkk\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624805 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.624846 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624839 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624878 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625077 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624880 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.625340 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.624653 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625638 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.625613 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.625734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.625658 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.627932 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.627902 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.628040 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.627972 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.691702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.691667 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.692660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.692636 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9jkk\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\") pod \"e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:43.725285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:43.725247 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:44.161053 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.161025 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:30:44.163190 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:30:44.163162 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10eef2d0_2f72_4671_9ae0_440d0bd219f4.slice/crio-53a8a2259d2c03fd13be9bc4f01ce7c3e1504e434e36c43ebbd6bd8e3abe30e6 WatchSource:0}: Error finding container 53a8a2259d2c03fd13be9bc4f01ce7c3e1504e434e36c43ebbd6bd8e3abe30e6: Status 404 returned error can't find the container with id 53a8a2259d2c03fd13be9bc4f01ce7c3e1504e434e36c43ebbd6bd8e3abe30e6 Nov 03 18:30:44.165501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.165470 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:44.165607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.165536 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:44.165607 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.165584 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:30:44.519093 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:30:44.519059 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:30:44.553282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.553251 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerStarted","Data":"fd5eab9ec7e0d66a465ea7fb9f6080b0ee98cd6299c30584ab88237b8c195e22"} Nov 03 18:30:44.553461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.553305 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerStarted","Data":"53a8a2259d2c03fd13be9bc4f01ce7c3e1504e434e36c43ebbd6bd8e3abe30e6"} Nov 03 18:30:44.677808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.677760 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podStartSLOduration=1.677747015 podStartE2EDuration="1.677747015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:44.67705219 +0000 UTC m=+3424.506135793" watchObservedRunningTime="2026-11-03 18:30:44.677747015 +0000 UTC m=+3424.506830616" Nov 03 18:30:44.726147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:44.726109 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:45.730767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:45.730737 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:46.563941 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:46.563900 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:46.564838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:46.564817 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:30:51.683218 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.683171 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:30:51.683917 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.683894 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb758195-1b8e-4b07-81a0-a40406583712" containerName="cleanup" Nov 03 18:30:51.687428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.687399 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.739234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.739204 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:30:51.786970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.786941 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ca6dbf\"" Nov 03 18:30:51.787134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.787035 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-302b5b77-dockercfg-crf2v\"" Nov 03 18:30:51.813487 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.813457 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.813669 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.813531 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gh6b\" (UniqueName: \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.853623 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.853589 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:30:51.857213 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.857191 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:51.914596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.914542 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.914773 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.914640 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gh6b\" (UniqueName: \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.915310 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.915286 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:51.918858 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.918833 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:30:51.978089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:51.978058 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gh6b\" (UniqueName: \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\") pod \"payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:52.009454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.009425 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:52.015809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.015782 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd4mv\" (UniqueName: \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.015927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.015821 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.117454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.117418 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd4mv\" (UniqueName: \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.117690 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.117486 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.118306 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.118288 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.182328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.182288 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd4mv\" (UniqueName: \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\") pod \"payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.386880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.386854 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:30:52.389404 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:30:52.389376 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod260ae723_65de_4626_9a34_b4d8ddcf8f9a.slice/crio-78ef28fb824410344c446374915a070fba46b713238fc9c8926795a68f9ab761 WatchSource:0}: Error finding container 78ef28fb824410344c446374915a070fba46b713238fc9c8926795a68f9ab761: Status 404 returned error can't find the container with id 78ef28fb824410344c446374915a070fba46b713238fc9c8926795a68f9ab761 Nov 03 18:30:52.470744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.470713 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:52.602405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.602364 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" event={"ID":"260ae723-65de-4626-9a34-b4d8ddcf8f9a","Type":"ContainerStarted","Data":"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813"} Nov 03 18:30:52.602596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.602433 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" event={"ID":"260ae723-65de-4626-9a34-b4d8ddcf8f9a","Type":"ContainerStarted","Data":"78ef28fb824410344c446374915a070fba46b713238fc9c8926795a68f9ab761"} Nov 03 18:30:52.602596 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.602562 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:30:52.722483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.722431 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" podStartSLOduration=1.722416714 podStartE2EDuration="1.722416714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:52.721282192 +0000 UTC m=+3432.550365794" watchObservedRunningTime="2026-11-03 18:30:52.722416714 +0000 UTC m=+3432.551500316" Nov 03 18:30:52.856202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:52.856090 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:30:52.860972 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:30:52.860940 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7733bdc_ad16_4158_8e79_87f6339416a1.slice/crio-ef80dd254d8dbe7e8b39e0f8f18c1dcfcf8831ee36d0b765c74a1e0993f0b999 WatchSource:0}: Error finding container ef80dd254d8dbe7e8b39e0f8f18c1dcfcf8831ee36d0b765c74a1e0993f0b999: Status 404 returned error can't find the container with id ef80dd254d8dbe7e8b39e0f8f18c1dcfcf8831ee36d0b765c74a1e0993f0b999 Nov 03 18:30:53.609428 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:53.609381 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" event={"ID":"f7733bdc-ad16-4158-8e79-87f6339416a1","Type":"ContainerStarted","Data":"d18d648567cb8264677ec7bf4cca809fd69c45d7e531d4c5de4b8ab3d2a4e0ad"} Nov 03 18:30:53.609652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:53.609474 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" event={"ID":"f7733bdc-ad16-4158-8e79-87f6339416a1","Type":"ContainerStarted","Data":"ef80dd254d8dbe7e8b39e0f8f18c1dcfcf8831ee36d0b765c74a1e0993f0b999"} Nov 03 18:30:53.729712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:53.729652 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" podStartSLOduration=2.7296381800000002 podStartE2EDuration="2.72963818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:30:53.728002134 +0000 UTC m=+3433.557085737" watchObservedRunningTime="2026-11-03 18:30:53.72963818 +0000 UTC m=+3433.558721781" Nov 03 18:30:54.615299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:54.615265 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:30:56.057831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:56.057794 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr"] Nov 03 18:30:56.107792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:56.107760 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-cd9jr"] Nov 03 18:30:56.293828 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:30:56.293799 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cf26149-7df2-4580-a396-d66c8934117a" path="/var/lib/kubelet/pods/6cf26149-7df2-4580-a396-d66c8934117a/volumes" Nov 03 18:31:03.612677 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:03.612638 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:31:05.623797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:05.623759 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:31:06.078608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:06.077067 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5"] Nov 03 18:31:06.084269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:06.084232 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-4qnz5"] Nov 03 18:31:06.294326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:06.294289 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cd441e2-4bc0-4404-9704-11fbd3714d0d" path="/var/lib/kubelet/pods/6cd441e2-4bc0-4404-9704-11fbd3714d0d/volumes" Nov 03 18:31:11.731286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.731252 4980 generic.go:356] "Generic (PLEG): container finished" podID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerID="fd5eab9ec7e0d66a465ea7fb9f6080b0ee98cd6299c30584ab88237b8c195e22" exitCode=0 Nov 03 18:31:11.731881 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.731332 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerDied","Data":"fd5eab9ec7e0d66a465ea7fb9f6080b0ee98cd6299c30584ab88237b8c195e22"} Nov 03 18:31:11.731960 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.731888 4980 scope.go:122] "RemoveContainer" containerID="fd5eab9ec7e0d66a465ea7fb9f6080b0ee98cd6299c30584ab88237b8c195e22" Nov 03 18:31:11.733270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.733244 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:31:11.733362 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.733317 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:31:11.733362 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:11.733353 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:31:12.739493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:12.739452 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerStarted","Data":"0a862cac46304453875a579b00f62f078a6170bf94175fb78bf5e332e5a55a05"} Nov 03 18:31:13.726011 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:13.725971 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:31:13.727321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:13.727297 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 18:31:13.727391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:13.727360 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 18:31:14.520345 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:31:14.520291 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:31:14.725691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:14.725658 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 18:31:14.725897 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:14.725730 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 18:31:15.726110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:15.726072 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 18:31:15.726600 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:15.726161 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 18:31:16.564723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:16.564677 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:31:16.725768 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:16.725731 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 18:31:16.725951 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:16.725799 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 18:31:17.726506 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:17.726467 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" start-of-body= Nov 03 18:31:17.726967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:17.726538 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.187:15021/healthz/ready\": dial tcp 10.132.0.187:15021: connect: connection refused" Nov 03 18:31:18.730508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:18.730471 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:31:18.731528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:18.731511 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:31:42.694807 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:42.694773 4980 scope.go:122] "RemoveContainer" containerID="1080af2dd245835842d924517a9d7c2345ae9eb8437449d226d3a89e99b07f22" Nov 03 18:31:42.704020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:42.704001 4980 scope.go:122] "RemoveContainer" containerID="cdf457e888d4ae3444739e80859dbe77773d4cd4305675a9fc5f108fa969d2d7" Nov 03 18:31:44.064939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:44.064900 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh"] Nov 03 18:31:44.114685 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:44.114651 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-nmfnh"] Nov 03 18:31:44.292448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:31:44.292410 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16f99b26-28b5-4536-b409-271a410b741a" path="/var/lib/kubelet/pods/16f99b26-28b5-4536-b409-271a410b741a/volumes" Nov 03 18:31:44.520228 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:31:44.520196 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:32:01.289556 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.289521 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl"] Nov 03 18:32:01.294557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.294540 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.353078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.353047 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl"] Nov 03 18:32:01.376230 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.376191 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.376444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.376267 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjxxk\" (UniqueName: \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.376444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.376362 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.399809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.399772 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:32:01.399809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.399786 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:32:01.477508 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.477473 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.477702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.477550 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.477702 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.477638 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjxxk\" (UniqueName: \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.478202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.478178 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.480089 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.480069 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.541103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.541038 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjxxk\" (UniqueName: \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\") pod \"maas-api-revoke-sub-8abfacaa-dfqwl\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.608624 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.608595 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:01.986307 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:01.986277 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl"] Nov 03 18:32:01.988398 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:32:01.988369 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda476cd34_4c7f_4bba_96c5_21ccac7e4516.slice/crio-b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7 WatchSource:0}: Error finding container b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7: Status 404 returned error can't find the container with id b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7 Nov 03 18:32:02.026134 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:02.026105 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" event={"ID":"a476cd34-4c7f-4bba-96c5-21ccac7e4516","Type":"ContainerStarted","Data":"b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7"} Nov 03 18:32:03.033628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:03.033592 4980 generic.go:356] "Generic (PLEG): container finished" podID="a476cd34-4c7f-4bba-96c5-21ccac7e4516" containerID="5f9724870b5c0cd37536851b8461c8df880be14a6e9219673886da3417402ac7" exitCode=0 Nov 03 18:32:03.034107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:03.033642 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" event={"ID":"a476cd34-4c7f-4bba-96c5-21ccac7e4516","Type":"ContainerDied","Data":"5f9724870b5c0cd37536851b8461c8df880be14a6e9219673886da3417402ac7"} Nov 03 18:32:04.188874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.188851 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:04.307044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.306945 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\" (UniqueName: \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\") pod \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " Nov 03 18:32:04.307044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.306997 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\") pod \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " Nov 03 18:32:04.307351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.307129 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\") pod \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\" (UID: \"a476cd34-4c7f-4bba-96c5-21ccac7e4516\") " Nov 03 18:32:04.307864 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.307839 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca" pod "a476cd34-4c7f-4bba-96c5-21ccac7e4516" (UID: "a476cd34-4c7f-4bba-96c5-21ccac7e4516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:04.310349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.310313 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca" pod "a476cd34-4c7f-4bba-96c5-21ccac7e4516" (UID: "a476cd34-4c7f-4bba-96c5-21ccac7e4516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:32:04.310349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.310326 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk" pod "a476cd34-4c7f-4bba-96c5-21ccac7e4516" (UID: "a476cd34-4c7f-4bba-96c5-21ccac7e4516"). InnerVolumeSpecName "kube-api-access-fjxxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:04.408694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.408664 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjxxk\" (UniqueName: \"kubernetes.io/projected/a476cd34-4c7f-4bba-96c5-21ccac7e4516-kube-api-access-fjxxk\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:04.408865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.408764 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:04.408865 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:04.408783 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a476cd34-4c7f-4bba-96c5-21ccac7e4516-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:05.045930 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:05.045896 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" event={"ID":"a476cd34-4c7f-4bba-96c5-21ccac7e4516","Type":"ContainerDied","Data":"b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7"} Nov 03 18:32:05.046116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:05.045950 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7a54029408dcefd2f236ef79afa11a8a44b0fab84698c970c27d0936d9e07c7" Nov 03 18:32:05.046116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:05.045909 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl" Nov 03 18:32:13.538716 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.538682 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4"] Nov 03 18:32:13.539186 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.539171 4980 cpu_manager.go:414] "containerMap: removing container" podUID="a476cd34-4c7f-4bba-96c5-21ccac7e4516" containerName="revoke-keys" Nov 03 18:32:13.539237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.539188 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a476cd34-4c7f-4bba-96c5-21ccac7e4516" containerName="revoke-keys" Nov 03 18:32:13.539311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.539299 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="a476cd34-4c7f-4bba-96c5-21ccac7e4516" containerName="revoke-keys" Nov 03 18:32:13.542379 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.542364 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.609709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.609669 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wzl7\" (UniqueName: \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.609908 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.609825 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.609908 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.609869 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.645585 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.645551 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:32:13.645767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.645595 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:32:13.648076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.648050 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4"] Nov 03 18:32:13.711415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.711376 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.711621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.711429 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.711621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.711545 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wzl7\" (UniqueName: \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.712223 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.712144 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.713991 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.713967 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.774533 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.774501 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wzl7\" (UniqueName: \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\") pod \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:13.855481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:13.855400 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:14.235096 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:14.235072 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4"] Nov 03 18:32:14.237137 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:32:14.237110 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e7df243_8670_4103_9250_3d2ddbc006a2.slice/crio-8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb WatchSource:0}: Error finding container 8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb: Status 404 returned error can't find the container with id 8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb Nov 03 18:32:14.520544 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:32:14.520468 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:32:15.101421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:15.101383 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" event={"ID":"7e7df243-8670-4103-9250-3d2ddbc006a2","Type":"ContainerStarted","Data":"723bb41be2870ca633df35a185c4b605e0bd8fafb946cb6ebbf1c17d6e51e344"} Nov 03 18:32:15.101981 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:15.101448 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" event={"ID":"7e7df243-8670-4103-9250-3d2ddbc006a2","Type":"ContainerStarted","Data":"8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb"} Nov 03 18:32:15.223023 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:15.222960 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" podStartSLOduration=2.2229393379999998 podStartE2EDuration="2.222939338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:32:15.22120089 +0000 UTC m=+3515.050284494" watchObservedRunningTime="2026-11-03 18:32:15.222939338 +0000 UTC m=+3515.052022942" Nov 03 18:32:16.057164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:16.057127 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn"] Nov 03 18:32:16.081226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:16.081195 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-dxvzn"] Nov 03 18:32:16.109525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:16.109489 4980 generic.go:356] "Generic (PLEG): container finished" podID="7e7df243-8670-4103-9250-3d2ddbc006a2" containerID="723bb41be2870ca633df35a185c4b605e0bd8fafb946cb6ebbf1c17d6e51e344" exitCode=0 Nov 03 18:32:16.109926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:16.109590 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" event={"ID":"7e7df243-8670-4103-9250-3d2ddbc006a2","Type":"ContainerDied","Data":"723bb41be2870ca633df35a185c4b605e0bd8fafb946cb6ebbf1c17d6e51e344"} Nov 03 18:32:16.292591 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:16.292535 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74d7b09-b818-4d64-b01f-2f97248336ca" path="/var/lib/kubelet/pods/a74d7b09-b818-4d64-b01f-2f97248336ca/volumes" Nov 03 18:32:17.249437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.249408 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:17.367372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.367289 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\" (UniqueName: \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\") pod \"7e7df243-8670-4103-9250-3d2ddbc006a2\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " Nov 03 18:32:17.367372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.367337 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\") pod \"7e7df243-8670-4103-9250-3d2ddbc006a2\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " Nov 03 18:32:17.367372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.367374 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\") pod \"7e7df243-8670-4103-9250-3d2ddbc006a2\" (UID: \"7e7df243-8670-4103-9250-3d2ddbc006a2\") " Nov 03 18:32:17.367797 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.367775 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca" pod "7e7df243-8670-4103-9250-3d2ddbc006a2" (UID: "7e7df243-8670-4103-9250-3d2ddbc006a2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:17.370161 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.370131 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca" pod "7e7df243-8670-4103-9250-3d2ddbc006a2" (UID: "7e7df243-8670-4103-9250-3d2ddbc006a2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:32:17.370282 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.370172 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7" pod "7e7df243-8670-4103-9250-3d2ddbc006a2" (UID: "7e7df243-8670-4103-9250-3d2ddbc006a2"). InnerVolumeSpecName "kube-api-access-4wzl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:17.468820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.468782 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:17.468820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.468814 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wzl7\" (UniqueName: \"kubernetes.io/projected/7e7df243-8670-4103-9250-3d2ddbc006a2-kube-api-access-4wzl7\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:17.468820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:17.468824 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e7df243-8670-4103-9250-3d2ddbc006a2-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:18.121925 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:18.121895 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:18.122151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:18.121940 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" event={"ID":"7e7df243-8670-4103-9250-3d2ddbc006a2","Type":"ContainerDied","Data":"8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb"} Nov 03 18:32:18.122151 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:18.121985 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8c8a0edd4d1ec6b6159b333dff99c8cb00d8be1ebff777a5576774c9a96f66cb" Nov 03 18:32:25.058056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:25.058021 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:32:25.168320 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:25.168282 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:32:30.058986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:30.058940 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" containerID="cri-o://3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813" gracePeriod=25 Nov 03 18:32:30.169434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:30.169377 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" containerID="cri-o://d18d648567cb8264677ec7bf4cca809fd69c45d7e531d4c5de4b8ab3d2a4e0ad" gracePeriod=25 Nov 03 18:32:30.285037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:30.285002 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4"] Nov 03 18:32:30.334913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:30.334833 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4"] Nov 03 18:32:30.340177 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:32:30.340148 4980 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4\" is forbidden: User \"system:node:ip-10-0-139-234.us-west-1.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-139-234.us-west-1.compute.internal' and this object" podUID="7e7df243-8670-4103-9250-3d2ddbc006a2" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-ca6dbf-7fhh4" Nov 03 18:32:32.293284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:32.293246 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e7df243-8670-4103-9250-3d2ddbc006a2" path="/var/lib/kubelet/pods/7e7df243-8670-4103-9250-3d2ddbc006a2/volumes" Nov 03 18:32:33.611412 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:33.611372 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:32:35.621883 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:35.621842 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:32:38.169295 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:38.169262 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:32:38.169715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:38.169462 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" containerID="cri-o://0a862cac46304453875a579b00f62f078a6170bf94175fb78bf5e332e5a55a05" gracePeriod=30 Nov 03 18:32:39.904229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.904181 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt"] Nov 03 18:32:39.905006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.904982 4980 cpu_manager.go:414] "containerMap: removing container" podUID="7e7df243-8670-4103-9250-3d2ddbc006a2" containerName="revoke-keys" Nov 03 18:32:39.905065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.905010 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e7df243-8670-4103-9250-3d2ddbc006a2" containerName="revoke-keys" Nov 03 18:32:39.905152 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.905139 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e7df243-8670-4103-9250-3d2ddbc006a2" containerName="revoke-keys" Nov 03 18:32:39.909871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.909850 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:39.967949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.967922 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt"] Nov 03 18:32:39.988876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.988851 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:39.989025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.988893 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:39.989025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:39.988928 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-697w8\" (UniqueName: \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.015034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.015003 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:32:40.015283 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.015266 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:32:40.090454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.090425 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.090641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.090471 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.090641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.090511 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-697w8\" (UniqueName: \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.091100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.091078 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.093006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.092982 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.154163 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.154129 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-697w8\" (UniqueName: \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\") pod \"maas-api-revoke-sub-3d4156f1-xm2mt\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.222557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.222526 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:40.607990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:40.607961 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt"] Nov 03 18:32:40.610102 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:32:40.610067 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3dad4f17_8243_4007_a2f0_069a6ef2f2cb.slice/crio-45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b WatchSource:0}: Error finding container 45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b: Status 404 returned error can't find the container with id 45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b Nov 03 18:32:41.279370 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:41.279332 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" event={"ID":"3dad4f17-8243-4007-a2f0-069a6ef2f2cb","Type":"ContainerStarted","Data":"cfe919d0ccfb766685d207b2933aa5ab958b2f145d527ec7eeeb062ce9374886"} Nov 03 18:32:41.279918 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:41.279393 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" event={"ID":"3dad4f17-8243-4007-a2f0-069a6ef2f2cb","Type":"ContainerStarted","Data":"45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b"} Nov 03 18:32:41.398547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:41.398494 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" podStartSLOduration=2.398479033 podStartE2EDuration="2.398479033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:32:41.397496748 +0000 UTC m=+3541.226580348" watchObservedRunningTime="2026-11-03 18:32:41.398479033 +0000 UTC m=+3541.227562634" Nov 03 18:32:42.286772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:42.286740 4980 generic.go:356] "Generic (PLEG): container finished" podID="3dad4f17-8243-4007-a2f0-069a6ef2f2cb" containerID="cfe919d0ccfb766685d207b2933aa5ab958b2f145d527ec7eeeb062ce9374886" exitCode=0 Nov 03 18:32:42.292313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:42.292284 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" event={"ID":"3dad4f17-8243-4007-a2f0-069a6ef2f2cb","Type":"ContainerDied","Data":"cfe919d0ccfb766685d207b2933aa5ab958b2f145d527ec7eeeb062ce9374886"} Nov 03 18:32:42.733344 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:42.733313 4980 scope.go:122] "RemoveContainer" containerID="1fbc128b8cc42f21fffc6c0da06042d519c4335d20710f4c7782e22fbdde2eee" Nov 03 18:32:42.742627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:42.742604 4980 scope.go:122] "RemoveContainer" containerID="a9fc487f39f8d3473427086af4bb4a96714173c42c05a0a5b7a11b97c0185cb7" Nov 03 18:32:42.751133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:42.751116 4980 scope.go:122] "RemoveContainer" containerID="78074ec859999ef3409dc9b2c80e118f4d3fb42859c8e32b3dea67edaacfb14a" Nov 03 18:32:43.293553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.293521 4980 generic.go:356] "Generic (PLEG): container finished" podID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerID="0a862cac46304453875a579b00f62f078a6170bf94175fb78bf5e332e5a55a05" exitCode=0 Nov 03 18:32:43.294010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.293619 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerDied","Data":"0a862cac46304453875a579b00f62f078a6170bf94175fb78bf5e332e5a55a05"} Nov 03 18:32:43.294010 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.293697 4980 scope.go:122] "RemoveContainer" containerID="fd5eab9ec7e0d66a465ea7fb9f6080b0ee98cd6299c30584ab88237b8c195e22" Nov 03 18:32:43.435916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.435858 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:32:43.465537 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.465512 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:43.527675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527647 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.527823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527687 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.527823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527721 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\" (UniqueName: \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\") pod \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " Nov 03 18:32:43.527823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527747 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.527823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527776 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.527823 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527815 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527925 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527962 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\") pod \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.527970 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528005 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\") pod \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\" (UID: \"3dad4f17-8243-4007-a2f0-069a6ef2f2cb\") " Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528048 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528079 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:32:43.528121 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528107 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528189 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528221 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\") pod \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\" (UID: \"10eef2d0-2f72-4671-9ae0-440d0bd219f4\") " Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528326 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca" pod "3dad4f17-8243-4007-a2f0-069a6ef2f2cb" (UID: "3dad4f17-8243-4007-a2f0-069a6ef2f2cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528385 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528596 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528667 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528685 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528698 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528712 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528727 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.528856 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.528740 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.530437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.530411 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:32:43.530437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.530412 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8" pod "3dad4f17-8243-4007-a2f0-069a6ef2f2cb" (UID: "3dad4f17-8243-4007-a2f0-069a6ef2f2cb"). InnerVolumeSpecName "kube-api-access-697w8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:43.530734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.530717 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca" pod "3dad4f17-8243-4007-a2f0-069a6ef2f2cb" (UID: "3dad4f17-8243-4007-a2f0-069a6ef2f2cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:32:43.530790 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.530743 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:43.531205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.531179 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "kube-api-access-c9jkk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:43.531205 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.531182 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo" pod "10eef2d0-2f72-4671-9ae0-440d0bd219f4" (UID: "10eef2d0-2f72-4671-9ae0-440d0bd219f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:32:43.611651 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.611610 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:32:43.630004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.629977 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-697w8\" (UniqueName: \"kubernetes.io/projected/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-kube-api-access-697w8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.630004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.630005 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.630182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.630016 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.630182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.630025 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3dad4f17-8243-4007-a2f0-069a6ef2f2cb-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.630182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.630034 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9jkk\" (UniqueName: \"kubernetes.io/projected/10eef2d0-2f72-4671-9ae0-440d0bd219f4-kube-api-access-c9jkk\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:43.630182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:43.630042 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10eef2d0-2f72-4671-9ae0-440d0bd219f4-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:44.301204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.301170 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" Nov 03 18:32:44.301708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.301198 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt" event={"ID":"3dad4f17-8243-4007-a2f0-069a6ef2f2cb","Type":"ContainerDied","Data":"45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b"} Nov 03 18:32:44.301708 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.301244 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="45acb834753a544dbf5b216c5673a63ee4319d9156695598de1c84e68245837b" Nov 03 18:32:44.302982 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.302960 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" Nov 03 18:32:44.303097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.302998 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc" event={"ID":"10eef2d0-2f72-4671-9ae0-440d0bd219f4","Type":"ContainerDied","Data":"53a8a2259d2c03fd13be9bc4f01ce7c3e1504e434e36c43ebbd6bd8e3abe30e6"} Nov 03 18:32:44.303097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.303050 4980 scope.go:122] "RemoveContainer" containerID="0a862cac46304453875a579b00f62f078a6170bf94175fb78bf5e332e5a55a05" Nov 03 18:32:44.519214 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:32:44.519141 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:32:44.527503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.527472 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:32:44.535445 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:44.535420 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ca6dbf-openshift-default-79c5c56d5-kkwzc"] Nov 03 18:32:45.621803 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:45.621766 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:32:46.295116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:46.295080 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" path="/var/lib/kubelet/pods/10eef2d0-2f72-4671-9ae0-440d0bd219f4/volumes" Nov 03 18:32:50.059694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.059657 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz"] Nov 03 18:32:50.109820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.109791 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-g2rjz"] Nov 03 18:32:50.294054 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.294025 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d14d393-d8bd-458e-83d3-155c080ac688" path="/var/lib/kubelet/pods/5d14d393-d8bd-458e-83d3-155c080ac688/volumes" Nov 03 18:32:50.314097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.314027 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:32:50.346247 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.346215 4980 generic.go:356] "Generic (PLEG): container finished" podID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerID="3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813" exitCode=0 Nov 03 18:32:50.346443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.346262 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" Nov 03 18:32:50.346443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.346293 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" event={"ID":"260ae723-65de-4626-9a34-b4d8ddcf8f9a","Type":"ContainerDied","Data":"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813"} Nov 03 18:32:50.346443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.346356 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6" event={"ID":"260ae723-65de-4626-9a34-b4d8ddcf8f9a","Type":"ContainerDied","Data":"78ef28fb824410344c446374915a070fba46b713238fc9c8926795a68f9ab761"} Nov 03 18:32:50.346443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.346385 4980 scope.go:122] "RemoveContainer" containerID="3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813" Nov 03 18:32:50.353423 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.353397 4980 generic.go:356] "Generic (PLEG): container finished" podID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerID="d18d648567cb8264677ec7bf4cca809fd69c45d7e531d4c5de4b8ab3d2a4e0ad" exitCode=0 Nov 03 18:32:50.353601 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.353445 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" event={"ID":"f7733bdc-ad16-4158-8e79-87f6339416a1","Type":"ContainerDied","Data":"d18d648567cb8264677ec7bf4cca809fd69c45d7e531d4c5de4b8ab3d2a4e0ad"} Nov 03 18:32:50.365526 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.365503 4980 scope.go:122] "RemoveContainer" containerID="3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813" Nov 03 18:32:50.365953 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:32:50.365917 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813\": container with ID starting with 3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813 not found: ID does not exist" containerID="3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813" Nov 03 18:32:50.366069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.365968 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813"} err="failed to get container status \"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813\": rpc error: code = NotFound desc = could not find container \"3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813\": container with ID starting with 3175c2f07e32e30788fd3452d8cda27521cae5afefa32aeff78b286c3807d813 not found: ID does not exist" Nov 03 18:32:50.396017 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.395987 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\" (UniqueName: \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\") pod \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " Nov 03 18:32:50.396182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.396154 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\") pod \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\" (UID: \"260ae723-65de-4626-9a34-b4d8ddcf8f9a\") " Nov 03 18:32:50.396799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.396763 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume" pod "260ae723-65de-4626-9a34-b4d8ddcf8f9a" (UID: "260ae723-65de-4626-9a34-b4d8ddcf8f9a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:50.398445 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.398414 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b" pod "260ae723-65de-4626-9a34-b4d8ddcf8f9a" (UID: "260ae723-65de-4626-9a34-b4d8ddcf8f9a"). InnerVolumeSpecName "kube-api-access-9gh6b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:50.425595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.425553 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:32:50.497271 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.497237 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\" (UniqueName: \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\") pod \"f7733bdc-ad16-4158-8e79-87f6339416a1\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " Nov 03 18:32:50.497429 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.497344 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\") pod \"f7733bdc-ad16-4158-8e79-87f6339416a1\" (UID: \"f7733bdc-ad16-4158-8e79-87f6339416a1\") " Nov 03 18:32:50.497785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.497763 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/260ae723-65de-4626-9a34-b4d8ddcf8f9a-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:50.497852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.497791 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9gh6b\" (UniqueName: \"kubernetes.io/projected/260ae723-65de-4626-9a34-b4d8ddcf8f9a-kube-api-access-9gh6b\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:50.497852 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.497796 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume" pod "f7733bdc-ad16-4158-8e79-87f6339416a1" (UID: "f7733bdc-ad16-4158-8e79-87f6339416a1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:32:50.499525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.499505 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv" pod "f7733bdc-ad16-4158-8e79-87f6339416a1" (UID: "f7733bdc-ad16-4158-8e79-87f6339416a1"). InnerVolumeSpecName "kube-api-access-kd4mv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:32:50.598924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.598850 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kd4mv\" (UniqueName: \"kubernetes.io/projected/f7733bdc-ad16-4158-8e79-87f6339416a1-kube-api-access-kd4mv\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:50.598924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.598878 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f7733bdc-ad16-4158-8e79-87f6339416a1-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:32:50.771090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.771048 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:32:50.776728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:50.776701 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ca6dbf-75c5cb9cd7j9g6"] Nov 03 18:32:51.360106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:51.360030 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" Nov 03 18:32:51.360106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:51.360067 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f" event={"ID":"f7733bdc-ad16-4158-8e79-87f6339416a1","Type":"ContainerDied","Data":"ef80dd254d8dbe7e8b39e0f8f18c1dcfcf8831ee36d0b765c74a1e0993f0b999"} Nov 03 18:32:51.360695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:51.360125 4980 scope.go:122] "RemoveContainer" containerID="d18d648567cb8264677ec7bf4cca809fd69c45d7e531d4c5de4b8ab3d2a4e0ad" Nov 03 18:32:51.528064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:51.528025 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:32:51.577785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:51.577757 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ca6dbf-868c7ggx8f"] Nov 03 18:32:52.293216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:52.293169 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" path="/var/lib/kubelet/pods/260ae723-65de-4626-9a34-b4d8ddcf8f9a/volumes" Nov 03 18:32:52.293712 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:32:52.293690 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" path="/var/lib/kubelet/pods/f7733bdc-ad16-4158-8e79-87f6339416a1/volumes" Nov 03 18:33:14.058197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:14.058159 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q"] Nov 03 18:33:14.108229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:14.108197 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-n9c8q"] Nov 03 18:33:14.293581 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:14.293534 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="068041b2-1944-4608-a9ff-2341a089b6f5" path="/var/lib/kubelet/pods/068041b2-1944-4608-a9ff-2341a089b6f5/volumes" Nov 03 18:33:14.520217 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:33:14.520180 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:33:41.210788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:41.210762 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:33:41.215612 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:41.215588 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:33:41.267468 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:41.267443 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:33:41.272552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:41.272532 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:33:42.779073 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:33:42.779036 4980 scope.go:122] "RemoveContainer" containerID="b6c984c60185bd7975920a2030eaa42e82160dfa30dbe3acb7d4870285dfd60a" Nov 03 18:33:44.519708 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:33:44.519672 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:34:14.520561 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:34:14.520521 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:34:42.814352 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:42.814301 4980 scope.go:122] "RemoveContainer" containerID="1958ac71c90feddaacab1bae165d18c7692eda2fe3cbb0c2f641561a26520432" Nov 03 18:34:42.868360 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:42.868333 4980 scope.go:122] "RemoveContainer" containerID="9f44674bacafad9ff913c1053ae6f9df3a949c1f761060cc19acf0b95b338de4" Nov 03 18:34:42.984993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:42.978403 4980 scope.go:122] "RemoveContainer" containerID="3ea1fcc23c18693da68d43a030a17ef70d6eaaa31bd64d05f7b717d7dd1e2bf8" Nov 03 18:34:43.630709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:43.630670 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-r96r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:34:43.631004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:43.630973 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-r96r9" podUID="250bf139-b14a-4c76-8bf3-842f5be6494f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.163:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:34:43.667597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:43.666676 4980 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-5df48fcc99-4cj7c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Nov 03 18:34:43.667597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:43.666742 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-5df48fcc99-4cj7c" podUID="819d1443-6698-423f-911d-8cb12ea7a3c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Nov 03 18:34:44.519025 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:34:44.518993 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:34:45.533595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.533529 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm"] Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.533967 4980 cpu_manager.go:414] "containerMap: removing container" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.533981 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534005 4980 cpu_manager.go:414] "containerMap: removing container" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534010 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534028 4980 cpu_manager.go:414] "containerMap: removing container" podUID="3dad4f17-8243-4007-a2f0-069a6ef2f2cb" containerName="revoke-keys" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534034 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dad4f17-8243-4007-a2f0-069a6ef2f2cb" containerName="revoke-keys" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534042 4980 cpu_manager.go:414] "containerMap: removing container" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534046 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534062 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" Nov 03 18:34:45.534103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534067 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" Nov 03 18:34:45.534441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534120 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7733bdc-ad16-4158-8e79-87f6339416a1" containerName="payload-pre-processing" Nov 03 18:34:45.534441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534131 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:34:45.534441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534140 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="260ae723-65de-4626-9a34-b4d8ddcf8f9a" containerName="payload-processing" Nov 03 18:34:45.534441 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.534148 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dad4f17-8243-4007-a2f0-069a6ef2f2cb" containerName="revoke-keys" Nov 03 18:34:45.538530 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.538512 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.597270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.597240 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm"] Nov 03 18:34:45.629979 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.629950 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.630116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.629992 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.630116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.630093 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnbxz\" (UniqueName: \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.644656 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.644632 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:34:45.644770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.644729 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:34:45.730968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.730931 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.731132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.730997 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnbxz\" (UniqueName: \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.731132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.731105 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.731550 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.731529 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.733583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.733546 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.793988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.793930 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnbxz\" (UniqueName: \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\") pod \"maas-api-revoke-sub-d576e68b-268rm\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:45.850845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:45.850814 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:46.232427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:46.232401 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm"] Nov 03 18:34:46.234751 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:34:46.234721 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbfac166_ab1a_4f4a_9eef_afd0de1346cd.slice/crio-dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979 WatchSource:0}: Error finding container dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979: Status 404 returned error can't find the container with id dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979 Nov 03 18:34:47.064598 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:47.062819 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz"] Nov 03 18:34:47.071687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:47.071647 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-cw5nz"] Nov 03 18:34:47.072923 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:47.072895 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" event={"ID":"cbfac166-ab1a-4f4a-9eef-afd0de1346cd","Type":"ContainerStarted","Data":"98008ecce625fe507360b4a6fe0e7af918610c1b9f41079db1e5391918653a6d"} Nov 03 18:34:47.073056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:47.072967 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" event={"ID":"cbfac166-ab1a-4f4a-9eef-afd0de1346cd","Type":"ContainerStarted","Data":"dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979"} Nov 03 18:34:47.196411 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:47.196359 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" podStartSLOduration=2.196345398 podStartE2EDuration="2.196345398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:34:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:34:47.195286065 +0000 UTC m=+3667.024369668" watchObservedRunningTime="2026-11-03 18:34:47.196345398 +0000 UTC m=+3667.025429000" Nov 03 18:34:48.081416 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:48.081382 4980 generic.go:356] "Generic (PLEG): container finished" podID="cbfac166-ab1a-4f4a-9eef-afd0de1346cd" containerID="98008ecce625fe507360b4a6fe0e7af918610c1b9f41079db1e5391918653a6d" exitCode=0 Nov 03 18:34:48.081851 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:48.081464 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" event={"ID":"cbfac166-ab1a-4f4a-9eef-afd0de1346cd","Type":"ContainerDied","Data":"98008ecce625fe507360b4a6fe0e7af918610c1b9f41079db1e5391918653a6d"} Nov 03 18:34:48.293976 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:48.293944 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64f171bb-d7e6-4e0a-8ae3-77b4c44c2386" path="/var/lib/kubelet/pods/64f171bb-d7e6-4e0a-8ae3-77b4c44c2386/volumes" Nov 03 18:34:49.232016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.231992 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:49.363245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.363173 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\") pod \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " Nov 03 18:34:49.363395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.363291 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\") pod \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " Nov 03 18:34:49.363395 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.363378 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\" (UniqueName: \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\") pod \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\" (UID: \"cbfac166-ab1a-4f4a-9eef-afd0de1346cd\") " Nov 03 18:34:49.363595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.363557 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca" pod "cbfac166-ab1a-4f4a-9eef-afd0de1346cd" (UID: "cbfac166-ab1a-4f4a-9eef-afd0de1346cd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:34:49.363774 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.363753 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:34:49.365649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.365619 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca" pod "cbfac166-ab1a-4f4a-9eef-afd0de1346cd" (UID: "cbfac166-ab1a-4f4a-9eef-afd0de1346cd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:34:49.365649 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.365623 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz" pod "cbfac166-ab1a-4f4a-9eef-afd0de1346cd" (UID: "cbfac166-ab1a-4f4a-9eef-afd0de1346cd"). InnerVolumeSpecName "kube-api-access-fnbxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:34:49.464748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.464714 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:34:49.464748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:49.464740 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnbxz\" (UniqueName: \"kubernetes.io/projected/cbfac166-ab1a-4f4a-9eef-afd0de1346cd-kube-api-access-fnbxz\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:34:50.092270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:50.092243 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" Nov 03 18:34:50.092464 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:50.092237 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm" event={"ID":"cbfac166-ab1a-4f4a-9eef-afd0de1346cd","Type":"ContainerDied","Data":"dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979"} Nov 03 18:34:50.092464 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:50.092367 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd74b5df313346062a23bcce338cd9c9d04dfebb8af3bf5d9cca11a403e5c979" Nov 03 18:34:58.064720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:58.064679 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p"] Nov 03 18:34:58.115140 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:58.115109 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-72b5p"] Nov 03 18:34:58.292371 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:34:58.292339 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69923fc6-93de-448f-8abf-592d3666e9d1" path="/var/lib/kubelet/pods/69923fc6-93de-448f-8abf-592d3666e9d1/volumes" Nov 03 18:35:14.520047 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:35:14.520006 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:35:14.673511 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.673473 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh"] Nov 03 18:35:14.673933 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.673920 4980 cpu_manager.go:414] "containerMap: removing container" podUID="cbfac166-ab1a-4f4a-9eef-afd0de1346cd" containerName="revoke-keys" Nov 03 18:35:14.673984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.673935 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbfac166-ab1a-4f4a-9eef-afd0de1346cd" containerName="revoke-keys" Nov 03 18:35:14.674023 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.674006 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="10eef2d0-2f72-4671-9ae0-440d0bd219f4" containerName="istio-proxy" Nov 03 18:35:14.674023 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.674017 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbfac166-ab1a-4f4a-9eef-afd0de1346cd" containerName="revoke-keys" Nov 03 18:35:14.677107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.677091 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.730788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.730756 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rchjr\" (UniqueName: \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.730972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.730893 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.730972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.730931 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.731873 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.731852 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:35:14.780563 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.780475 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:35:14.782670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.782647 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh"] Nov 03 18:35:14.831840 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.831804 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rchjr\" (UniqueName: \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.832013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.831880 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.832013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.831908 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.832513 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.832494 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.834503 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.834476 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.894990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.894957 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rchjr\" (UniqueName: \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\") pod \"maas-api-revoke-sub-d2bc93bc-2l7fh\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:14.990067 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:14.990032 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:15.375132 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:15.375017 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh"] Nov 03 18:35:15.377729 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:35:15.377695 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e2a0c6d_0300_41d1_8978_2c04a3ae5851.slice/crio-4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2 WatchSource:0}: Error finding container 4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2: Status 404 returned error can't find the container with id 4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2 Nov 03 18:35:16.258590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:16.258542 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" event={"ID":"9e2a0c6d-0300-41d1-8978-2c04a3ae5851","Type":"ContainerStarted","Data":"f8ed310a2ee8bec8356654935884dcd3d1a2aaae5d2e44a20f3802da289919ff"} Nov 03 18:35:16.258968 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:16.258625 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" event={"ID":"9e2a0c6d-0300-41d1-8978-2c04a3ae5851","Type":"ContainerStarted","Data":"4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2"} Nov 03 18:35:16.377637 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:16.377560 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" podStartSLOduration=2.377544825 podStartE2EDuration="2.377544825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:16.377082745 +0000 UTC m=+3696.206166355" watchObservedRunningTime="2026-11-03 18:35:16.377544825 +0000 UTC m=+3696.206628421" Nov 03 18:35:17.265246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:17.265210 4980 generic.go:356] "Generic (PLEG): container finished" podID="9e2a0c6d-0300-41d1-8978-2c04a3ae5851" containerID="f8ed310a2ee8bec8356654935884dcd3d1a2aaae5d2e44a20f3802da289919ff" exitCode=0 Nov 03 18:35:17.265770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:17.265271 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" event={"ID":"9e2a0c6d-0300-41d1-8978-2c04a3ae5851","Type":"ContainerDied","Data":"f8ed310a2ee8bec8356654935884dcd3d1a2aaae5d2e44a20f3802da289919ff"} Nov 03 18:35:18.421181 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.421154 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:18.569992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.569907 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\" (UniqueName: \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\") pod \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " Nov 03 18:35:18.570183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.570026 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\") pod \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " Nov 03 18:35:18.570183 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.570049 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\") pod \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\" (UID: \"9e2a0c6d-0300-41d1-8978-2c04a3ae5851\") " Nov 03 18:35:18.570520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.570484 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca" pod "9e2a0c6d-0300-41d1-8978-2c04a3ae5851" (UID: "9e2a0c6d-0300-41d1-8978-2c04a3ae5851"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:35:18.572159 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.572130 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr" pod "9e2a0c6d-0300-41d1-8978-2c04a3ae5851" (UID: "9e2a0c6d-0300-41d1-8978-2c04a3ae5851"). InnerVolumeSpecName "kube-api-access-rchjr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:35:18.572334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.572311 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca" pod "9e2a0c6d-0300-41d1-8978-2c04a3ae5851" (UID: "9e2a0c6d-0300-41d1-8978-2c04a3ae5851"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:35:18.671627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.671554 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:35:18.671627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.671621 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:35:18.671627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:18.671635 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rchjr\" (UniqueName: \"kubernetes.io/projected/9e2a0c6d-0300-41d1-8978-2c04a3ae5851-kube-api-access-rchjr\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:35:19.276259 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:19.276232 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" Nov 03 18:35:19.276444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:19.276232 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh" event={"ID":"9e2a0c6d-0300-41d1-8978-2c04a3ae5851","Type":"ContainerDied","Data":"4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2"} Nov 03 18:35:19.276444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:19.276353 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4259df61798b9c6b0ea98fc24214403277bf77875264cd9be0027785e18b97b2" Nov 03 18:35:27.785894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.785854 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:35:27.786314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.786277 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9e2a0c6d-0300-41d1-8978-2c04a3ae5851" containerName="revoke-keys" Nov 03 18:35:27.786314 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.786288 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e2a0c6d-0300-41d1-8978-2c04a3ae5851" containerName="revoke-keys" Nov 03 18:35:27.786405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.786363 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e2a0c6d-0300-41d1-8978-2c04a3ae5851" containerName="revoke-keys" Nov 03 18:35:27.790083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.790061 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.844830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.844788 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-057118-openshift-default-dockercfg-cfg7r\"" Nov 03 18:35:27.897540 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.897486 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:35:27.975341 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975305 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975352 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975397 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975611 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975681 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975730 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975780 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsddm\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.975882 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975835 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:27.976094 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:27.975917 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077285 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077414 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077445 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077489 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077539 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077566 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077646 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077678 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsddm\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077715 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.077808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077757 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.078102 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.077846 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.078169 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.078153 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.078451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.078424 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.078451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.078428 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.080356 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.080333 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.080448 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.080370 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.140995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.140963 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.141157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.141106 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsddm\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\") pod \"e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.404443 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.404358 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:28.837627 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.837588 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:35:28.873624 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:35:28.873592 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9abdd94c_ddf2_4cab_9a4b_d5ae50040415.slice/crio-4cb289cf9063ccb22fd327b03df36cf913955392db76baa4f525a08515f60e3c WatchSource:0}: Error finding container 4cb289cf9063ccb22fd327b03df36cf913955392db76baa4f525a08515f60e3c: Status 404 returned error can't find the container with id 4cb289cf9063ccb22fd327b03df36cf913955392db76baa4f525a08515f60e3c Nov 03 18:35:28.875935 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.875900 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:28.876006 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.875976 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:28.876044 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:28.876005 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:29.340686 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:29.340643 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerStarted","Data":"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18"} Nov 03 18:35:29.340928 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:29.340723 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerStarted","Data":"4cb289cf9063ccb22fd327b03df36cf913955392db76baa4f525a08515f60e3c"} Nov 03 18:35:29.404752 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:29.404712 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:29.410170 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:29.410143 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:29.466367 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:29.466300 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podStartSLOduration=2.466286472 podStartE2EDuration="2.466286472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:29.464689089 +0000 UTC m=+3709.293772691" watchObservedRunningTime="2026-11-03 18:35:29.466286472 +0000 UTC m=+3709.295370073" Nov 03 18:35:30.345885 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:30.345848 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:30.346990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:30.346970 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:36.308389 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.308352 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:35:36.316436 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.316406 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.353826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.353791 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:35:36.360002 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.359973 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.360179 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.360017 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdhsm\" (UniqueName: \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.375060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.375022 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-057118\"" Nov 03 18:35:36.375501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.375481 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-cdc7c41f-dockercfg-vdrl9\"" Nov 03 18:35:36.461611 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.461558 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.461784 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.461642 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdhsm\" (UniqueName: \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.462262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.462241 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.467937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.467913 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:35:36.471670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.471654 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.525111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.525078 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdhsm\" (UniqueName: \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\") pod \"payload-processing-e2e-worker-single-main-057118-7759775565l2zl\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.532239 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.532214 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:35:36.562780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.562687 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.562780 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.562754 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k76qw\" (UniqueName: \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.646418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.646383 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:36.664297 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.664259 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.665262 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.665234 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k76qw\" (UniqueName: \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.665396 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.665153 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.728340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.728304 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k76qw\" (UniqueName: \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\") pod \"payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:36.785130 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:36.785090 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:37.027244 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.027091 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:35:37.029686 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:35:37.029662 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ee2cb8_4bb5_40ed_a1a4_8c401db3aa0d.slice/crio-1777277fef01f814694c05fbfc3228e1b8c6f7101c892452ef3e17ceff878157 WatchSource:0}: Error finding container 1777277fef01f814694c05fbfc3228e1b8c6f7101c892452ef3e17ceff878157: Status 404 returned error can't find the container with id 1777277fef01f814694c05fbfc3228e1b8c6f7101c892452ef3e17ceff878157 Nov 03 18:35:37.162773 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.162552 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:35:37.164987 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:35:37.164956 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ae6d296_6d19_44bf_b5cd_928c52025981.slice/crio-4d0cbec05a51ebf72cedf1cc7902b4ba393a9cdb8ad047de22436ddfb3ef07f8 WatchSource:0}: Error finding container 4d0cbec05a51ebf72cedf1cc7902b4ba393a9cdb8ad047de22436ddfb3ef07f8: Status 404 returned error can't find the container with id 4d0cbec05a51ebf72cedf1cc7902b4ba393a9cdb8ad047de22436ddfb3ef07f8 Nov 03 18:35:37.384458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.384422 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" event={"ID":"6ae6d296-6d19-44bf-b5cd-928c52025981","Type":"ContainerStarted","Data":"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606"} Nov 03 18:35:37.384905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.384480 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" event={"ID":"6ae6d296-6d19-44bf-b5cd-928c52025981","Type":"ContainerStarted","Data":"4d0cbec05a51ebf72cedf1cc7902b4ba393a9cdb8ad047de22436ddfb3ef07f8"} Nov 03 18:35:37.384905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.384614 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:37.386075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.386053 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" event={"ID":"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d","Type":"ContainerStarted","Data":"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48"} Nov 03 18:35:37.386185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.386089 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" event={"ID":"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d","Type":"ContainerStarted","Data":"1777277fef01f814694c05fbfc3228e1b8c6f7101c892452ef3e17ceff878157"} Nov 03 18:35:37.386483 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.386468 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:37.504710 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.504660 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" podStartSLOduration=1.504646017 podStartE2EDuration="1.504646017s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:37.502753188 +0000 UTC m=+3717.331836794" watchObservedRunningTime="2026-11-03 18:35:37.504646017 +0000 UTC m=+3717.333729619" Nov 03 18:35:37.636498 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:37.633153 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" podStartSLOduration=1.633130661 podStartE2EDuration="1.633130661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:35:37.630091803 +0000 UTC m=+3717.459175406" watchObservedRunningTime="2026-11-03 18:35:37.633130661 +0000 UTC m=+3717.462214264" Nov 03 18:35:43.322809 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:43.322765 4980 scope.go:122] "RemoveContainer" containerID="30f19b95feb0a68274179fe6515b32280aa051e8fe8439c8d16dae31cdbf3df9" Nov 03 18:35:44.519693 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:35:44.519659 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:35:48.394713 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:48.394681 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:35:49.400613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:49.400558 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:35:57.528051 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.528019 4980 generic.go:356] "Generic (PLEG): container finished" podID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerID="3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18" exitCode=0 Nov 03 18:35:57.528519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.528095 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerDied","Data":"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18"} Nov 03 18:35:57.528565 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.528529 4980 scope.go:122] "RemoveContainer" containerID="3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18" Nov 03 18:35:57.529845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.529816 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:57.529916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.529884 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:57.529955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:57.529917 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:35:58.536033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:58.535997 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerStarted","Data":"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179"} Nov 03 18:35:59.404978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:59.404944 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:35:59.406064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:59.406039 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 18:35:59.406127 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:35:59.406098 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 18:36:00.346460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:00.346425 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:36:00.404996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:00.404954 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 18:36:00.405208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:00.405013 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 18:36:01.405334 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:01.405296 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 18:36:01.405788 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:01.405357 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 18:36:02.405311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:02.405274 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 18:36:02.405523 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:02.405333 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 18:36:03.404917 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:03.404874 4980 patch_prober.go:29] interesting pod/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" start-of-body= Nov 03 18:36:03.405109 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:03.404937 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.195:15021/healthz/ready\": dial tcp 10.132.0.195:15021: connect: connection refused" Nov 03 18:36:04.409887 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:04.409853 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:36:04.411025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:04.411003 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:36:14.519337 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:36:14.519293 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:36:43.346670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:43.346636 4980 scope.go:122] "RemoveContainer" containerID="8b85d57f9b131a629ba7a616630d85463669ece3a0a8068d9f0285401d9ea593" Nov 03 18:36:43.355555 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:43.355530 4980 scope.go:122] "RemoveContainer" containerID="5a9ec6b523cedb91a8bcecea20a1271399a267163720b574ce67b2572920046e" Nov 03 18:36:44.518747 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:36:44.518710 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:36:45.968792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:45.968756 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj"] Nov 03 18:36:45.972557 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:45.972539 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.026862 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.026831 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:36:46.075855 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.075822 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:36:46.078020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.077996 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj"] Nov 03 18:36:46.106110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.106083 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.106248 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.106138 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts4dr\" (UniqueName: \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.106309 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.106286 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.207826 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.207796 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts4dr\" (UniqueName: \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.207970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.207879 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.208015 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.207965 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.208622 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.208598 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.210227 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.210202 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.270421 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.270363 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts4dr\" (UniqueName: \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\") pod \"maas-api-revoke-sub-cd97c760-76mxj\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.287787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.287763 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:46.665368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.665342 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj"] Nov 03 18:36:46.667503 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:36:46.667475 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0e5351_c948_4a00_abaa_d868a519925e.slice/crio-408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731 WatchSource:0}: Error finding container 408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731: Status 404 returned error can't find the container with id 408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731 Nov 03 18:36:46.795328 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:46.795295 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" event={"ID":"1e0e5351-c948-4a00-abaa-d868a519925e","Type":"ContainerStarted","Data":"408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731"} Nov 03 18:36:47.805660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:47.805552 4980 generic.go:356] "Generic (PLEG): container finished" podID="1e0e5351-c948-4a00-abaa-d868a519925e" containerID="c5da74577a0b0c5451e30ab72bf3a23e3decf70f5829ff6ecf912a445301233d" exitCode=0 Nov 03 18:36:47.805660 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:47.805616 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" event={"ID":"1e0e5351-c948-4a00-abaa-d868a519925e","Type":"ContainerDied","Data":"c5da74577a0b0c5451e30ab72bf3a23e3decf70f5829ff6ecf912a445301233d"} Nov 03 18:36:49.007645 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.007617 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:49.046106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.046074 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\") pod \"1e0e5351-c948-4a00-abaa-d868a519925e\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " Nov 03 18:36:49.046270 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.046120 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\" (UniqueName: \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\") pod \"1e0e5351-c948-4a00-abaa-d868a519925e\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " Nov 03 18:36:49.046318 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.046258 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\") pod \"1e0e5351-c948-4a00-abaa-d868a519925e\" (UID: \"1e0e5351-c948-4a00-abaa-d868a519925e\") " Nov 03 18:36:49.046636 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.046610 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca" pod "1e0e5351-c948-4a00-abaa-d868a519925e" (UID: "1e0e5351-c948-4a00-abaa-d868a519925e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:36:49.046763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.046697 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:36:49.048993 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.048974 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca" pod "1e0e5351-c948-4a00-abaa-d868a519925e" (UID: "1e0e5351-c948-4a00-abaa-d868a519925e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:36:49.049074 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.048998 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr" pod "1e0e5351-c948-4a00-abaa-d868a519925e" (UID: "1e0e5351-c948-4a00-abaa-d868a519925e"). InnerVolumeSpecName "kube-api-access-ts4dr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:36:49.148319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.148222 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e0e5351-c948-4a00-abaa-d868a519925e-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:36:49.148319 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.148262 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts4dr\" (UniqueName: \"kubernetes.io/projected/1e0e5351-c948-4a00-abaa-d868a519925e-kube-api-access-ts4dr\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:36:49.826880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.826851 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" Nov 03 18:36:49.827076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.826884 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj" event={"ID":"1e0e5351-c948-4a00-abaa-d868a519925e","Type":"ContainerDied","Data":"408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731"} Nov 03 18:36:49.827076 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:49.826929 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="408343ebd0b47981973ba4fe3d6c0a2028921d56f348cb3cc83f2d1b256e3731" Nov 03 18:36:57.624936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.624901 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82"] Nov 03 18:36:57.625386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.625342 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1e0e5351-c948-4a00-abaa-d868a519925e" containerName="revoke-keys" Nov 03 18:36:57.625386 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.625354 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e0e5351-c948-4a00-abaa-d868a519925e" containerName="revoke-keys" Nov 03 18:36:57.625471 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.625432 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e0e5351-c948-4a00-abaa-d868a519925e" containerName="revoke-keys" Nov 03 18:36:57.628629 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.628613 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.727983 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.727951 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqjxn\" (UniqueName: \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.728142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.727991 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.728142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.728094 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.731419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.731400 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:36:57.731481 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.731441 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:36:57.733770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.733752 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82"] Nov 03 18:36:57.829031 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.828994 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqjxn\" (UniqueName: \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.829246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.829039 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.829246 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.829082 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.829988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.829958 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.831831 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.831813 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.892133 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.892065 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqjxn\" (UniqueName: \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\") pod \"maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:57.941020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:57.940995 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:36:58.324466 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:58.324442 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82"] Nov 03 18:36:58.326732 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:36:58.326704 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1778e1a8_d090_4799_99c4_5fb4753a4598.slice/crio-877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004 WatchSource:0}: Error finding container 877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004: Status 404 returned error can't find the container with id 877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004 Nov 03 18:36:58.875104 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:58.875065 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" event={"ID":"1778e1a8-d090-4799-99c4-5fb4753a4598","Type":"ContainerStarted","Data":"b4e63439282f8c2d49f5a9665a09b92dd1f8ad66979566c8b8cd56fba89d135d"} Nov 03 18:36:58.875757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:58.875117 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" event={"ID":"1778e1a8-d090-4799-99c4-5fb4753a4598","Type":"ContainerStarted","Data":"877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004"} Nov 03 18:36:58.993108 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:58.993046 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" podStartSLOduration=1.993028265 podStartE2EDuration="1.993028265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:36:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:36:58.992834511 +0000 UTC m=+3798.821918114" watchObservedRunningTime="2026-11-03 18:36:58.993028265 +0000 UTC m=+3798.822111867" Nov 03 18:36:59.883341 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:59.883305 4980 generic.go:356] "Generic (PLEG): container finished" podID="1778e1a8-d090-4799-99c4-5fb4753a4598" containerID="b4e63439282f8c2d49f5a9665a09b92dd1f8ad66979566c8b8cd56fba89d135d" exitCode=0 Nov 03 18:36:59.883859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:36:59.883400 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" event={"ID":"1778e1a8-d090-4799-99c4-5fb4753a4598","Type":"ContainerDied","Data":"b4e63439282f8c2d49f5a9665a09b92dd1f8ad66979566c8b8cd56fba89d135d"} Nov 03 18:37:01.038910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.038885 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:37:01.063197 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.063175 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\") pod \"1778e1a8-d090-4799-99c4-5fb4753a4598\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " Nov 03 18:37:01.063333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.063295 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\") pod \"1778e1a8-d090-4799-99c4-5fb4753a4598\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " Nov 03 18:37:01.063333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.063328 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\" (UniqueName: \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\") pod \"1778e1a8-d090-4799-99c4-5fb4753a4598\" (UID: \"1778e1a8-d090-4799-99c4-5fb4753a4598\") " Nov 03 18:37:01.063684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.063662 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca" pod "1778e1a8-d090-4799-99c4-5fb4753a4598" (UID: "1778e1a8-d090-4799-99c4-5fb4753a4598"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:37:01.065419 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.065396 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca" pod "1778e1a8-d090-4799-99c4-5fb4753a4598" (UID: "1778e1a8-d090-4799-99c4-5fb4753a4598"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:37:01.065551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.065531 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn" pod "1778e1a8-d090-4799-99c4-5fb4753a4598" (UID: "1778e1a8-d090-4799-99c4-5fb4753a4598"). InnerVolumeSpecName "kube-api-access-vqjxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:37:01.164947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.164872 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:01.164947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.164908 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqjxn\" (UniqueName: \"kubernetes.io/projected/1778e1a8-d090-4799-99c4-5fb4753a4598-kube-api-access-vqjxn\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:01.164947 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.164926 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1778e1a8-d090-4799-99c4-5fb4753a4598-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:01.895064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.895031 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" Nov 03 18:37:01.895064 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.895042 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82" event={"ID":"1778e1a8-d090-4799-99c4-5fb4753a4598","Type":"ContainerDied","Data":"877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004"} Nov 03 18:37:01.895286 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:01.895092 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="877f174474683f4f716ed5da2b10e150eaeedb1ee90daa36eab7fabb4c0b7004" Nov 03 18:37:05.058231 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:05.058199 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl"] Nov 03 18:37:05.108934 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:05.108903 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8abfacaa-dfqwl"] Nov 03 18:37:06.294736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:06.294699 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a476cd34-4c7f-4bba-96c5-21ccac7e4516" path="/var/lib/kubelet/pods/a476cd34-4c7f-4bba-96c5-21ccac7e4516/volumes" Nov 03 18:37:09.149238 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:09.149199 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:37:09.261138 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:09.261092 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:37:14.149899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:14.149858 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" containerID="cri-o://ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48" gracePeriod=25 Nov 03 18:37:14.262194 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:14.262149 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" containerID="cri-o://9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606" gracePeriod=25 Nov 03 18:37:14.519349 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:14.519304 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:37:14.559889 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:14.559851 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82"] Nov 03 18:37:14.609804 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:14.609769 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-057118-tpg82"] Nov 03 18:37:16.292904 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:16.292868 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1778e1a8-d090-4799-99c4-5fb4753a4598" path="/var/lib/kubelet/pods/1778e1a8-d090-4799-99c4-5fb4753a4598/volumes" Nov 03 18:37:18.393875 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:18.393838 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:37:19.399608 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:19.399549 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:37:22.388848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:22.388765 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:37:22.389339 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:22.388973 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" containerID="cri-o://a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179" gracePeriod=30 Nov 03 18:37:27.644014 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.643987 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:37:27.646185 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646156 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646229 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646259 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646307 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646335 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646391 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:37:27.646434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646409 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646464 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646493 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646564 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\") pod \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\" (UID: \"9abdd94c-ddf2-4cab-9a4b-d5ae50040415\") " Nov 03 18:37:27.646675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646661 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:37:27.646892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646704 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:37:27.646892 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646798 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:37:27.646988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.646913 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:37:27.647029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.647011 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.647029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.647025 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.647100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.647034 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.647100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.647043 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.647100 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.647051 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.648615 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.648562 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:37:27.648763 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.648748 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:37:27.648977 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.648958 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "kube-api-access-gsddm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:37:27.649082 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.649060 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo" pod "9abdd94c-ddf2-4cab-9a4b-d5ae50040415" (UID: "9abdd94c-ddf2-4cab-9a4b-d5ae50040415"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:37:27.747653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.747615 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.747653 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.747650 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.747853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.747663 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsddm\" (UniqueName: \"kubernetes.io/projected/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-kube-api-access-gsddm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:27.747853 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:27.747673 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9abdd94c-ddf2-4cab-9a4b-d5ae50040415-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:28.051092 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.051056 4980 generic.go:356] "Generic (PLEG): container finished" podID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerID="a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179" exitCode=0 Nov 03 18:37:28.051285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.051113 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" Nov 03 18:37:28.051285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.051142 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerDied","Data":"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179"} Nov 03 18:37:28.051285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.051225 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7" event={"ID":"9abdd94c-ddf2-4cab-9a4b-d5ae50040415","Type":"ContainerDied","Data":"4cb289cf9063ccb22fd327b03df36cf913955392db76baa4f525a08515f60e3c"} Nov 03 18:37:28.051285 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.051252 4980 scope.go:122] "RemoveContainer" containerID="a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179" Nov 03 18:37:28.061401 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.061383 4980 scope.go:122] "RemoveContainer" containerID="3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18" Nov 03 18:37:28.070110 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.070093 4980 scope.go:122] "RemoveContainer" containerID="a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179" Nov 03 18:37:28.070386 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:28.070361 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179\": container with ID starting with a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179 not found: ID does not exist" containerID="a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179" Nov 03 18:37:28.070500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.070392 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179"} err="failed to get container status \"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179\": rpc error: code = NotFound desc = could not find container \"a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179\": container with ID starting with a48f0d10f7ce0489adf0a853f8853289366b8430aab898ef7b1fc53b93cd3179 not found: ID does not exist" Nov 03 18:37:28.070500 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.070416 4980 scope.go:122] "RemoveContainer" containerID="3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18" Nov 03 18:37:28.070696 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:28.070678 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18\": container with ID starting with 3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18 not found: ID does not exist" containerID="3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18" Nov 03 18:37:28.070745 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.070706 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18"} err="failed to get container status \"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18\": rpc error: code = NotFound desc = could not find container \"3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18\": container with ID starting with 3a20c2e263d80ae41f9d88116ae2f94bcb353a15a90e148e0eabe7fa704d3c18 not found: ID does not exist" Nov 03 18:37:28.222137 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.222105 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:37:28.229258 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.229232 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-057118-openshift-default-86c567d9c7xkzz7"] Nov 03 18:37:28.292705 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.292679 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" path="/var/lib/kubelet/pods/9abdd94c-ddf2-4cab-9a4b-d5ae50040415/volumes" Nov 03 18:37:28.393617 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:28.393539 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:37:29.399438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:29.399403 4980 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Nov 03 18:37:34.435497 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.435468 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:37:34.520016 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.519983 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\") pod \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " Nov 03 18:37:34.520157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.520140 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\" (UniqueName: \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\") pod \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\" (UID: \"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d\") " Nov 03 18:37:34.520410 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.520382 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume" pod "f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" (UID: "f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:37:34.520525 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.520496 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:34.522307 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.522283 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm" pod "f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" (UID: "f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d"). InnerVolumeSpecName "kube-api-access-hdhsm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:37:34.527256 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.527232 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:37:34.620836 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.620803 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\") pod \"6ae6d296-6d19-44bf-b5cd-928c52025981\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " Nov 03 18:37:34.621039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.620939 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\" (UniqueName: \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\") pod \"6ae6d296-6d19-44bf-b5cd-928c52025981\" (UID: \"6ae6d296-6d19-44bf-b5cd-928c52025981\") " Nov 03 18:37:34.621202 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.621177 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume" pod "6ae6d296-6d19-44bf-b5cd-928c52025981" (UID: "6ae6d296-6d19-44bf-b5cd-928c52025981"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:37:34.621321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.621307 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ae6d296-6d19-44bf-b5cd-928c52025981-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:34.621359 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.621325 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdhsm\" (UniqueName: \"kubernetes.io/projected/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d-kube-api-access-hdhsm\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:34.623263 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.623234 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw" pod "6ae6d296-6d19-44bf-b5cd-928c52025981" (UID: "6ae6d296-6d19-44bf-b5cd-928c52025981"). InnerVolumeSpecName "kube-api-access-k76qw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:37:34.723059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:34.723021 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k76qw\" (UniqueName: \"kubernetes.io/projected/6ae6d296-6d19-44bf-b5cd-928c52025981-kube-api-access-k76qw\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:37:35.094524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.094154 4980 generic.go:356] "Generic (PLEG): container finished" podID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerID="9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606" exitCode=0 Nov 03 18:37:35.094524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.094192 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" Nov 03 18:37:35.094524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.094228 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" event={"ID":"6ae6d296-6d19-44bf-b5cd-928c52025981","Type":"ContainerDied","Data":"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606"} Nov 03 18:37:35.094524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.094292 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87" event={"ID":"6ae6d296-6d19-44bf-b5cd-928c52025981","Type":"ContainerDied","Data":"4d0cbec05a51ebf72cedf1cc7902b4ba393a9cdb8ad047de22436ddfb3ef07f8"} Nov 03 18:37:35.094524 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.094327 4980 scope.go:122] "RemoveContainer" containerID="9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606" Nov 03 18:37:35.096949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.096924 4980 generic.go:356] "Generic (PLEG): container finished" podID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerID="ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48" exitCode=0 Nov 03 18:37:35.100358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.097374 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" Nov 03 18:37:35.101085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.100843 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" event={"ID":"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d","Type":"ContainerDied","Data":"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48"} Nov 03 18:37:35.101085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.100895 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl" event={"ID":"f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d","Type":"ContainerDied","Data":"1777277fef01f814694c05fbfc3228e1b8c6f7101c892452ef3e17ceff878157"} Nov 03 18:37:35.115147 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.115071 4980 scope.go:122] "RemoveContainer" containerID="9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606" Nov 03 18:37:35.115454 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:35.115425 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606\": container with ID starting with 9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606 not found: ID does not exist" containerID="9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606" Nov 03 18:37:35.115547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.115468 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606"} err="failed to get container status \"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606\": rpc error: code = NotFound desc = could not find container \"9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606\": container with ID starting with 9eb52448e2a59e32721f3bf26b5da3b74f2c39f506a0dde03876a7f22e470606 not found: ID does not exist" Nov 03 18:37:35.115547 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.115496 4980 scope.go:122] "RemoveContainer" containerID="ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48" Nov 03 18:37:35.132465 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.132430 4980 scope.go:122] "RemoveContainer" containerID="ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48" Nov 03 18:37:35.132916 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:35.132881 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48\": container with ID starting with ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48 not found: ID does not exist" containerID="ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48" Nov 03 18:37:35.133020 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.132944 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48"} err="failed to get container status \"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48\": rpc error: code = NotFound desc = could not find container \"ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48\": container with ID starting with ff387de60d2489e9b42e2dfdbdd23a19c3fe75cab873a72218e2b54981bb9d48 not found: ID does not exist" Nov 03 18:37:35.264805 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.264756 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:37:35.315978 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.315944 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-057118-5fd54c5r87"] Nov 03 18:37:35.428038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.427934 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:37:35.434871 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:35.434837 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-057118-7759775565l2zl"] Nov 03 18:37:36.335618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:36.334773 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" path="/var/lib/kubelet/pods/6ae6d296-6d19-44bf-b5cd-928c52025981/volumes" Nov 03 18:37:36.335618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:36.335325 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" path="/var/lib/kubelet/pods/f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d/volumes" Nov 03 18:37:44.058304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:44.058252 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt"] Nov 03 18:37:44.108048 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:44.108013 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xm2mt"] Nov 03 18:37:44.300995 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:37:44.300960 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dad4f17-8243-4007-a2f0-069a6ef2f2cb" path="/var/lib/kubelet/pods/3dad4f17-8243-4007-a2f0-069a6ef2f2cb/volumes" Nov 03 18:37:44.519023 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:37:44.518986 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:38:14.519814 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:38:14.519770 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:38:41.245075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:41.245004 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:38:41.249909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:41.249888 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:38:41.304869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:41.304837 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:38:41.310738 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:41.310710 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:38:43.411434 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:43.411401 4980 scope.go:122] "RemoveContainer" containerID="723bb41be2870ca633df35a185c4b605e0bd8fafb946cb6ebbf1c17d6e51e344" Nov 03 18:38:43.420772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:43.420749 4980 scope.go:122] "RemoveContainer" containerID="cfe919d0ccfb766685d207b2933aa5ab958b2f145d527ec7eeeb062ce9374886" Nov 03 18:38:43.429370 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:43.429350 4980 scope.go:122] "RemoveContainer" containerID="5f9724870b5c0cd37536851b8461c8df880be14a6e9219673886da3417402ac7" Nov 03 18:38:44.520777 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:38:44.520736 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:38:45.983551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.983506 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6"] Nov 03 18:38:45.984101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984005 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1778e1a8-d090-4799-99c4-5fb4753a4598" containerName="revoke-keys" Nov 03 18:38:45.984101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984044 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1778e1a8-d090-4799-99c4-5fb4753a4598" containerName="revoke-keys" Nov 03 18:38:45.984101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984074 4980 cpu_manager.go:414] "containerMap: removing container" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" Nov 03 18:38:45.984101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984087 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984109 4980 cpu_manager.go:414] "containerMap: removing container" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984118 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984152 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984159 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984170 4980 cpu_manager.go:414] "containerMap: removing container" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984176 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984284 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1778e1a8-d090-4799-99c4-5fb4753a4598" containerName="revoke-keys" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984297 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ee2cb8-4bb5-40ed-a1a4-8c401db3aa0d" containerName="payload-processing" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984305 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae6d296-6d19-44bf-b5cd-928c52025981" containerName="payload-pre-processing" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984318 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.984349 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.984328 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="9abdd94c-ddf2-4cab-9a4b-d5ae50040415" containerName="istio-proxy" Nov 03 18:38:45.987514 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:45.987495 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.042180 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.042142 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:38:46.066792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.066749 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.066792 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.066796 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbv9p\" (UniqueName: \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.067038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.066916 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.090872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.090840 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:38:46.092927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.092901 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6"] Nov 03 18:38:46.168193 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.168156 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.168193 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.168194 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbv9p\" (UniqueName: \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.168446 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.168248 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.169105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.169019 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.170806 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.170783 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.231495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.231463 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbv9p\" (UniqueName: \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\") pod \"maas-api-revoke-sub-452b6d7b-h46t6\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.301543 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.301466 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:46.680948 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:46.680913 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6"] Nov 03 18:38:46.686861 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:38:46.686537 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68e73e89_bff3_4cd8_8a80_f6093ad05b61.slice/crio-0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd WatchSource:0}: Error finding container 0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd: Status 404 returned error can't find the container with id 0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd Nov 03 18:38:47.551243 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:47.551210 4980 generic.go:356] "Generic (PLEG): container finished" podID="68e73e89-bff3-4cd8-8a80-f6093ad05b61" containerID="257d7b4a3a5d7f79c32ed8d94092776cf90e41a8f70b3946836b7a3445a9f0f7" exitCode=0 Nov 03 18:38:47.551698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:47.551300 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" event={"ID":"68e73e89-bff3-4cd8-8a80-f6093ad05b61","Type":"ContainerDied","Data":"257d7b4a3a5d7f79c32ed8d94092776cf90e41a8f70b3946836b7a3445a9f0f7"} Nov 03 18:38:47.551698 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:47.551352 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" event={"ID":"68e73e89-bff3-4cd8-8a80-f6093ad05b61","Type":"ContainerStarted","Data":"0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd"} Nov 03 18:38:48.763718 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.763682 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:38:48.897561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.896636 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\") pod \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " Nov 03 18:38:48.897561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.896768 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\") pod \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " Nov 03 18:38:48.897561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.896886 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\" (UniqueName: \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\") pod \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\" (UID: \"68e73e89-bff3-4cd8-8a80-f6093ad05b61\") " Nov 03 18:38:48.897561 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.897372 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca" pod "68e73e89-bff3-4cd8-8a80-f6093ad05b61" (UID: "68e73e89-bff3-4cd8-8a80-f6093ad05b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:38:48.900743 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.900692 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p" pod "68e73e89-bff3-4cd8-8a80-f6093ad05b61" (UID: "68e73e89-bff3-4cd8-8a80-f6093ad05b61"). InnerVolumeSpecName "kube-api-access-dbv9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:38:48.915916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.915854 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca" pod "68e73e89-bff3-4cd8-8a80-f6093ad05b61" (UID: "68e73e89-bff3-4cd8-8a80-f6093ad05b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:38:48.998737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.998693 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:38:48.998737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.998738 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbv9p\" (UniqueName: \"kubernetes.io/projected/68e73e89-bff3-4cd8-8a80-f6093ad05b61-kube-api-access-dbv9p\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:38:48.999036 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:48.998754 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/68e73e89-bff3-4cd8-8a80-f6093ad05b61-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:38:49.594597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:49.592792 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" event={"ID":"68e73e89-bff3-4cd8-8a80-f6093ad05b61","Type":"ContainerDied","Data":"0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd"} Nov 03 18:38:49.594597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:49.592859 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca98c365ecce17cea725e50752c95f6b75fbd970992f71d899dd5f6b42596fd" Nov 03 18:38:49.594597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:38:49.592942 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6" Nov 03 18:39:14.520136 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:39:14.520047 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:39:22.119552 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.119514 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw"] Nov 03 18:39:22.120071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.120038 4980 cpu_manager.go:414] "containerMap: removing container" podUID="68e73e89-bff3-4cd8-8a80-f6093ad05b61" containerName="revoke-keys" Nov 03 18:39:22.120071 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.120060 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68e73e89-bff3-4cd8-8a80-f6093ad05b61" containerName="revoke-keys" Nov 03 18:39:22.120192 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.120182 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="68e73e89-bff3-4cd8-8a80-f6093ad05b61" containerName="revoke-keys" Nov 03 18:39:22.123383 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.123363 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.177876 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.177846 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:39:22.226640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.226607 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:39:22.228893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.228870 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw"] Nov 03 18:39:22.266331 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.266293 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm5pb\" (UniqueName: \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.266559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.266416 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.266559 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.266461 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.367059 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.367019 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm5pb\" (UniqueName: \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.367273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.367097 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.367273 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.367124 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.367874 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.367849 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.370065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.370011 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.430562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.430525 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm5pb\" (UniqueName: \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\") pod \"maas-api-revoke-sub-3e31a506-45mgw\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.437974 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.437945 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:22.825224 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:22.825175 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw"] Nov 03 18:39:22.827437 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:39:22.827404 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0b1e96c_c86b_456a_b3e5_3008409038ed.slice/crio-828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a WatchSource:0}: Error finding container 828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a: Status 404 returned error can't find the container with id 828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a Nov 03 18:39:23.829597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:23.829074 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" event={"ID":"c0b1e96c-c86b-456a-b3e5-3008409038ed","Type":"ContainerStarted","Data":"8d21e33a484287166a443e93e81354dd9e6e1ec93099c17863e47c674740d667"} Nov 03 18:39:23.829597 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:23.829147 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" event={"ID":"c0b1e96c-c86b-456a-b3e5-3008409038ed","Type":"ContainerStarted","Data":"828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a"} Nov 03 18:39:24.852748 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:24.852703 4980 generic.go:356] "Generic (PLEG): container finished" podID="c0b1e96c-c86b-456a-b3e5-3008409038ed" containerID="8d21e33a484287166a443e93e81354dd9e6e1ec93099c17863e47c674740d667" exitCode=0 Nov 03 18:39:24.853340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:24.852915 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" event={"ID":"c0b1e96c-c86b-456a-b3e5-3008409038ed","Type":"ContainerDied","Data":"8d21e33a484287166a443e93e81354dd9e6e1ec93099c17863e47c674740d667"} Nov 03 18:39:25.994453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:25.994429 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:26.104583 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.104551 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\" (UniqueName: \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\") pod \"c0b1e96c-c86b-456a-b3e5-3008409038ed\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " Nov 03 18:39:26.104757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.104665 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\") pod \"c0b1e96c-c86b-456a-b3e5-3008409038ed\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " Nov 03 18:39:26.104757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.104733 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\") pod \"c0b1e96c-c86b-456a-b3e5-3008409038ed\" (UID: \"c0b1e96c-c86b-456a-b3e5-3008409038ed\") " Nov 03 18:39:26.105080 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.105050 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca" pod "c0b1e96c-c86b-456a-b3e5-3008409038ed" (UID: "c0b1e96c-c86b-456a-b3e5-3008409038ed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:39:26.106910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.106886 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb" pod "c0b1e96c-c86b-456a-b3e5-3008409038ed" (UID: "c0b1e96c-c86b-456a-b3e5-3008409038ed"). InnerVolumeSpecName "kube-api-access-sm5pb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:39:26.106910 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.106899 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca" pod "c0b1e96c-c86b-456a-b3e5-3008409038ed" (UID: "c0b1e96c-c86b-456a-b3e5-3008409038ed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:39:26.206160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.206111 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:26.206160 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.206159 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm5pb\" (UniqueName: \"kubernetes.io/projected/c0b1e96c-c86b-456a-b3e5-3008409038ed-kube-api-access-sm5pb\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:26.206397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.206176 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0b1e96c-c86b-456a-b3e5-3008409038ed-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:26.864245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.864216 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" Nov 03 18:39:26.864426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.864210 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw" event={"ID":"c0b1e96c-c86b-456a-b3e5-3008409038ed","Type":"ContainerDied","Data":"828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a"} Nov 03 18:39:26.864426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:26.864342 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="828f7231a5cc55ae166d7a52325dd53425e84996867ee53d2da153eed8a50a7a" Nov 03 18:39:44.519963 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:39:44.519919 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:39:50.058715 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:50.058675 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm"] Nov 03 18:39:50.108697 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:50.108658 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-268rm"] Nov 03 18:39:50.292427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:50.292392 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbfac166-ab1a-4f4a-9eef-afd0de1346cd" path="/var/lib/kubelet/pods/cbfac166-ab1a-4f4a-9eef-afd0de1346cd/volumes" Nov 03 18:39:54.751079 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.751043 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx"] Nov 03 18:39:54.751501 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.751482 4980 cpu_manager.go:414] "containerMap: removing container" podUID="c0b1e96c-c86b-456a-b3e5-3008409038ed" containerName="revoke-keys" Nov 03 18:39:54.751560 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.751498 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0b1e96c-c86b-456a-b3e5-3008409038ed" containerName="revoke-keys" Nov 03 18:39:54.751655 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.751642 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0b1e96c-c86b-456a-b3e5-3008409038ed" containerName="revoke-keys" Nov 03 18:39:54.754939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.754920 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.802112 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.802074 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.802276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.802137 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7wxs\" (UniqueName: \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.802276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.802248 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.809418 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.809397 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Nov 03 18:39:54.858292 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.858258 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-lzw79\"" Nov 03 18:39:54.860642 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.860616 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx"] Nov 03 18:39:54.903843 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.903812 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.904024 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.903863 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7wxs\" (UniqueName: \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.904024 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.903892 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.904491 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.904471 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.906476 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.906455 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:54.967232 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:54.967202 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7wxs\" (UniqueName: \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\") pod \"maas-api-revoke-sub-b923ecf3-gt9fx\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:55.067740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:55.067672 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:55.448097 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:55.448072 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx"] Nov 03 18:39:55.450817 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:39:55.450774 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4224210_7b9e_4add_82c0_7f586cc643cc.slice/crio-cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5 WatchSource:0}: Error finding container cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5: Status 404 returned error can't find the container with id cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5 Nov 03 18:39:56.056400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:56.056305 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" event={"ID":"e4224210-7b9e-4add-82c0-7f586cc643cc","Type":"ContainerStarted","Data":"b14b667a354505cce6fdd666ea2731bd857ebd30c24154e3502deda543ec9e09"} Nov 03 18:39:56.056400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:56.056366 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" event={"ID":"e4224210-7b9e-4add-82c0-7f586cc643cc","Type":"ContainerStarted","Data":"cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5"} Nov 03 18:39:56.175438 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:56.175388 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" podStartSLOduration=2.175375092 podStartE2EDuration="2.175375092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:39:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:39:56.17362793 +0000 UTC m=+3976.002711533" watchObservedRunningTime="2026-11-03 18:39:56.175375092 +0000 UTC m=+3976.004458694" Nov 03 18:39:57.062992 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:57.062954 4980 generic.go:356] "Generic (PLEG): container finished" podID="e4224210-7b9e-4add-82c0-7f586cc643cc" containerID="b14b667a354505cce6fdd666ea2731bd857ebd30c24154e3502deda543ec9e09" exitCode=0 Nov 03 18:39:57.063458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:57.063039 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" event={"ID":"e4224210-7b9e-4add-82c0-7f586cc643cc","Type":"ContainerDied","Data":"b14b667a354505cce6fdd666ea2731bd857ebd30c24154e3502deda543ec9e09"} Nov 03 18:39:58.220095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.220068 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:58.340162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.340080 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\" (UniqueName: \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\") pod \"e4224210-7b9e-4add-82c0-7f586cc643cc\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " Nov 03 18:39:58.340321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.340198 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\") pod \"e4224210-7b9e-4add-82c0-7f586cc643cc\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " Nov 03 18:39:58.340321 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.340273 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\") pod \"e4224210-7b9e-4add-82c0-7f586cc643cc\" (UID: \"e4224210-7b9e-4add-82c0-7f586cc643cc\") " Nov 03 18:39:58.340859 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.340832 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca" pod "e4224210-7b9e-4add-82c0-7f586cc643cc" (UID: "e4224210-7b9e-4add-82c0-7f586cc643cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:39:58.343333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.343291 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs" pod "e4224210-7b9e-4add-82c0-7f586cc643cc" (UID: "e4224210-7b9e-4add-82c0-7f586cc643cc"). InnerVolumeSpecName "kube-api-access-x7wxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:39:58.343333 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.343321 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca" pod "e4224210-7b9e-4add-82c0-7f586cc643cc" (UID: "e4224210-7b9e-4add-82c0-7f586cc643cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:39:58.442139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.442103 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7wxs\" (UniqueName: \"kubernetes.io/projected/e4224210-7b9e-4add-82c0-7f586cc643cc-kube-api-access-x7wxs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:58.442139 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.442140 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-openshift-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:58.442415 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:58.442155 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4224210-7b9e-4add-82c0-7f586cc643cc-maas-api-xks-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:39:59.074190 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:59.074158 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" Nov 03 18:39:59.074397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:59.074189 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx" event={"ID":"e4224210-7b9e-4add-82c0-7f586cc643cc","Type":"ContainerDied","Data":"cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5"} Nov 03 18:39:59.074397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:39:59.074239 4980 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfe168c999cdfec884a648991899111e8822363b8f8ce616f1acf2c7a4df4df5" Nov 03 18:40:08.095528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.095497 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:08.095972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.095950 4980 cpu_manager.go:414] "containerMap: removing container" podUID="e4224210-7b9e-4add-82c0-7f586cc643cc" containerName="revoke-keys" Nov 03 18:40:08.095972 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.095963 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4224210-7b9e-4add-82c0-7f586cc643cc" containerName="revoke-keys" Nov 03 18:40:08.096060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.096043 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4224210-7b9e-4add-82c0-7f586cc643cc" containerName="revoke-keys" Nov 03 18:40:08.099444 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.099423 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.203382 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.203351 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Nov 03 18:40:08.203703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.203351 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-fhmsl\"" Nov 03 18:40:08.203703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.203350 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Nov 03 18:40:08.208101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.208078 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:08.237833 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.237808 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.237967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.237846 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4nf9\" (UniqueName: \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.237967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.237948 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.238038 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.237985 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.338937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.338903 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.338937 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.338937 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4nf9\" (UniqueName: \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.339154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.338989 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.339154 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.339015 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.339793 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.339726 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.341647 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.341625 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.341734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.341665 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.401946 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.401884 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4nf9\" (UniqueName: \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\") pod \"maas-api-6cc56655c-7grb2\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.413544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.413519 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:08.796301 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.796270 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:08.799011 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:40:08.798983 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f778882_34c7_4a55_af11_52686b6b5ea7.slice/crio-830fddecdf0e68b0a884777df86fae26f205403cd4d3a039dcf37e6859f8852a WatchSource:0}: Error finding container 830fddecdf0e68b0a884777df86fae26f205403cd4d3a039dcf37e6859f8852a: Status 404 returned error can't find the container with id 830fddecdf0e68b0a884777df86fae26f205403cd4d3a039dcf37e6859f8852a Nov 03 18:40:08.800493 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:08.800476 4980 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Nov 03 18:40:09.124365 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:09.124266 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" event={"ID":"1f778882-34c7-4a55-af11-52686b6b5ea7","Type":"ContainerStarted","Data":"830fddecdf0e68b0a884777df86fae26f205403cd4d3a039dcf37e6859f8852a"} Nov 03 18:40:10.132165 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:10.132119 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" event={"ID":"1f778882-34c7-4a55-af11-52686b6b5ea7","Type":"ContainerStarted","Data":"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659"} Nov 03 18:40:10.132592 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:10.132187 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:10.252590 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:10.252520 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" podStartSLOduration=1.365440246 podStartE2EDuration="2.252504796s" totalImagesPullingTime="887.06455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:10.25192046 +0000 UTC m=+3990.081004087" watchObservedRunningTime="2026-11-03 18:40:10.252504796 +0000 UTC m=+3990.081588398" Nov 03 18:40:14.520336 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:40:14.520278 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:40:15.280400 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.280365 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:40:15.284304 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.284280 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.369013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.368984 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-c188b2e5-openshift-default-dockercfg-6gwmv\"" Nov 03 18:40:15.407141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407108 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407141 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407144 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407167 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49t5j\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407333 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407380 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407367 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407396 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407439 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407499 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407456 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.407621 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.407527 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.421431 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.421402 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:40:15.509084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509053 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509129 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509145 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509168 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49t5j\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509221 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509245 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509274 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509314 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509364 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509338 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509640 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509698 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509739 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.509898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.509880 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.510245 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.510116 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.511728 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.511704 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.511949 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.511933 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.572625 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.572528 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.572986 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.572618 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49t5j\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\") pod \"e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:15.597388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:15.597362 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:16.035026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.034803 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:40:16.037425 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:40:16.037399 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359cf870_4cef_45f4_97cb_ba93daa30bce.slice/crio-e6a70acfdc54abe086684482c235428c3465c656e05a6f736e5113ec2b8e8c96 WatchSource:0}: Error finding container e6a70acfdc54abe086684482c235428c3465c656e05a6f736e5113ec2b8e8c96: Status 404 returned error can't find the container with id e6a70acfdc54abe086684482c235428c3465c656e05a6f736e5113ec2b8e8c96 Nov 03 18:40:16.039560 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.039529 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:16.039663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.039616 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:16.039663 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.039647 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:16.143545 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.143515 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:16.166379 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.166343 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerStarted","Data":"f9b3b7fe4380b0f6f6533504501bd90898e6c914f9740c0a344d4dc56e895a97"} Nov 03 18:40:16.166584 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.166411 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerStarted","Data":"e6a70acfdc54abe086684482c235428c3465c656e05a6f736e5113ec2b8e8c96"} Nov 03 18:40:16.364863 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.364784 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87"] Nov 03 18:40:16.368641 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.368620 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.390996 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.390940 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podStartSLOduration=1.390922007 podStartE2EDuration="1.390922007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:16.387911239 +0000 UTC m=+3996.216994841" watchObservedRunningTime="2026-11-03 18:40:16.390922007 +0000 UTC m=+3996.220005612" Nov 03 18:40:16.430437 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.430403 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87"] Nov 03 18:40:16.522799 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.522723 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-openshift-service-ca\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.523103 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.523053 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9gznv\" (UniqueName: \"kubernetes.io/projected/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-kube-api-access-9gznv\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.523234 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.523209 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-metrics-certs\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.523303 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.523280 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-maas-api-tls\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.597955 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.597922 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:16.624252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.624185 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-openshift-service-ca\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.624252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.624216 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9gznv\" (UniqueName: \"kubernetes.io/projected/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-kube-api-access-9gznv\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.624477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.624265 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-metrics-certs\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.624477 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.624304 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-maas-api-tls\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.624984 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.624964 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-openshift-service-ca\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.626911 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.626892 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-metrics-certs\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.627003 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.626933 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-maas-api-tls\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.687203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.687168 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9gznv\" (UniqueName: \"kubernetes.io/projected/bb3bbb76-a54e-4aa8-bd73-a35af88de3b1-kube-api-access-9gznv\") pod \"maas-api-7456bf56b5-ccl87\" (UID: \"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1\") " pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:16.985173 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:16.985138 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:17.363964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:17.363941 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87"] Nov 03 18:40:17.366177 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:40:17.366152 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3bbb76_a54e_4aa8_bd73_a35af88de3b1.slice/crio-34acf1e5e4477db575874e0e731a5e7fae33a568b1cdb9d86ac8e849a9ce8484 WatchSource:0}: Error finding container 34acf1e5e4477db575874e0e731a5e7fae33a568b1cdb9d86ac8e849a9ce8484: Status 404 returned error can't find the container with id 34acf1e5e4477db575874e0e731a5e7fae33a568b1cdb9d86ac8e849a9ce8484 Nov 03 18:40:17.602454 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:17.602424 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:18.178027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:18.178004 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" event={"ID":"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1","Type":"ContainerStarted","Data":"34acf1e5e4477db575874e0e731a5e7fae33a568b1cdb9d86ac8e849a9ce8484"} Nov 03 18:40:18.178460 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:18.178440 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:18.179370 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:18.179352 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:19.065105 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:19.065066 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh"] Nov 03 18:40:19.072980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:19.072943 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-2l7fh"] Nov 03 18:40:19.184998 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:19.184956 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" event={"ID":"bb3bbb76-a54e-4aa8-bd73-a35af88de3b1","Type":"ContainerStarted","Data":"f678123b85829b8a59e739bd50b0417f6b5849c0a093b4b362f94c61c2439c47"} Nov 03 18:40:19.185316 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:19.185294 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:19.306643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:19.306596 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" podStartSLOduration=2.4981495369999998 podStartE2EDuration="3.30658234s" totalImagesPullingTime="808.432803ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:40:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:40:19.304675646 +0000 UTC m=+3999.133759250" watchObservedRunningTime="2026-11-03 18:40:19.30658234 +0000 UTC m=+3999.135665932" Nov 03 18:40:20.292847 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:20.292772 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e2a0c6d-0300-41d1-8978-2c04a3ae5851" path="/var/lib/kubelet/pods/9e2a0c6d-0300-41d1-8978-2c04a3ae5851/volumes" Nov 03 18:40:25.196894 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.196866 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7456bf56b5-ccl87" Nov 03 18:40:25.371087 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.371053 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:25.371298 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.371274 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" podUID="1f778882-34c7-4a55-af11-52686b6b5ea7" containerName="maas-api" containerID="cri-o://f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659" gracePeriod=30 Nov 03 18:40:25.632084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.632053 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:25.713670 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.713644 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\") pod \"1f778882-34c7-4a55-af11-52686b6b5ea7\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " Nov 03 18:40:25.713827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.713689 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\") pod \"1f778882-34c7-4a55-af11-52686b6b5ea7\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " Nov 03 18:40:25.713827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.713723 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\" (UniqueName: \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\") pod \"1f778882-34c7-4a55-af11-52686b6b5ea7\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " Nov 03 18:40:25.713827 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.713793 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\") pod \"1f778882-34c7-4a55-af11-52686b6b5ea7\" (UID: \"1f778882-34c7-4a55-af11-52686b6b5ea7\") " Nov 03 18:40:25.714228 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.714193 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca" pod "1f778882-34c7-4a55-af11-52686b6b5ea7" (UID: "1f778882-34c7-4a55-af11-52686b6b5ea7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:40:25.715970 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.715945 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs" pod "1f778882-34c7-4a55-af11-52686b6b5ea7" (UID: "1f778882-34c7-4a55-af11-52686b6b5ea7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:25.716083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.715977 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9" pod "1f778882-34c7-4a55-af11-52686b6b5ea7" (UID: "1f778882-34c7-4a55-af11-52686b6b5ea7"). InnerVolumeSpecName "kube-api-access-h4nf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:40:25.716083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.716041 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls" pod "1f778882-34c7-4a55-af11-52686b6b5ea7" (UID: "1f778882-34c7-4a55-af11-52686b6b5ea7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Nov 03 18:40:25.815407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.815333 4980 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-maas-api-tls\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:40:25.815407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.815367 4980 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1f778882-34c7-4a55-af11-52686b6b5ea7-metrics-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:40:25.815407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.815386 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4nf9\" (UniqueName: \"kubernetes.io/projected/1f778882-34c7-4a55-af11-52686b6b5ea7-kube-api-access-h4nf9\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:40:25.815407 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:25.815402 4980 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f778882-34c7-4a55-af11-52686b6b5ea7-openshift-service-ca\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:40:26.225674 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.225643 4980 generic.go:356] "Generic (PLEG): container finished" podID="1f778882-34c7-4a55-af11-52686b6b5ea7" containerID="f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659" exitCode=0 Nov 03 18:40:26.226174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.225685 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" event={"ID":"1f778882-34c7-4a55-af11-52686b6b5ea7","Type":"ContainerDied","Data":"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659"} Nov 03 18:40:26.226174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.225701 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" Nov 03 18:40:26.226174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.225726 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6cc56655c-7grb2" event={"ID":"1f778882-34c7-4a55-af11-52686b6b5ea7","Type":"ContainerDied","Data":"830fddecdf0e68b0a884777df86fae26f205403cd4d3a039dcf37e6859f8852a"} Nov 03 18:40:26.226174 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.225747 4980 scope.go:122] "RemoveContainer" containerID="f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659" Nov 03 18:40:26.235478 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.235461 4980 scope.go:122] "RemoveContainer" containerID="f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659" Nov 03 18:40:26.235762 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:40:26.235744 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659\": container with ID starting with f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659 not found: ID does not exist" containerID="f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659" Nov 03 18:40:26.235880 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.235771 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659"} err="failed to get container status \"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659\": rpc error: code = NotFound desc = could not find container \"f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659\": container with ID starting with f6d48ceaaae8e9d9f75d7b58a129af9eb8f04a8b5bf720def942f6004e722659 not found: ID does not exist" Nov 03 18:40:26.394008 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.393976 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:26.443740 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:26.443710 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6cc56655c-7grb2"] Nov 03 18:40:28.292632 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:28.292601 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f778882-34c7-4a55-af11-52686b6b5ea7" path="/var/lib/kubelet/pods/1f778882-34c7-4a55-af11-52686b6b5ea7/volumes" Nov 03 18:40:42.329939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.329907 4980 generic.go:356] "Generic (PLEG): container finished" podID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerID="f9b3b7fe4380b0f6f6533504501bd90898e6c914f9740c0a344d4dc56e895a97" exitCode=0 Nov 03 18:40:42.330406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.329952 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerDied","Data":"f9b3b7fe4380b0f6f6533504501bd90898e6c914f9740c0a344d4dc56e895a97"} Nov 03 18:40:42.330406 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.330304 4980 scope.go:122] "RemoveContainer" containerID="f9b3b7fe4380b0f6f6533504501bd90898e6c914f9740c0a344d4dc56e895a97" Nov 03 18:40:42.331652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.331620 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:42.331723 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.331694 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:42.331767 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:42.331733 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:40:43.336848 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:43.336814 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerStarted","Data":"c30c6f54dd71ac24e9979539b7ddf3b7445cc015df4a9667d4c4403be9e938f3"} Nov 03 18:40:43.598695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:43.598560 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:43.599691 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:43.599668 4980 patch_prober.go:29] interesting pod/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 18:40:43.599829 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:43.599716 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 18:40:44.521136 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:40:44.521101 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:40:44.598237 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:44.598202 4980 patch_prober.go:29] interesting pod/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 18:40:44.598417 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:44.598258 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 18:40:45.598039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:45.598006 4980 patch_prober.go:29] interesting pod/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 18:40:45.598453 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:45.598058 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 18:40:46.597868 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:46.597831 4980 patch_prober.go:29] interesting pod/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 18:40:46.598099 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:46.597896 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 18:40:47.598107 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:47.598072 4980 patch_prober.go:29] interesting pod/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" start-of-body= Nov 03 18:40:47.598544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:47.598142 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.204:15021/healthz/ready\": dial tcp 10.132.0.204:15021: connect: connection refused" Nov 03 18:40:48.179602 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:48.178679 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:48.601519 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:48.601492 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:40:48.602766 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:40:48.602744 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:41:14.519438 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:41:14.519401 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:41:21.460462 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:21.460420 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:41:21.460893 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:21.460640 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" containerID="cri-o://c30c6f54dd71ac24e9979539b7ddf3b7445cc015df4a9667d4c4403be9e938f3" gracePeriod=30 Nov 03 18:41:23.205939 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.205899 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:41:23.206639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.206592 4980 cpu_manager.go:414] "containerMap: removing container" podUID="1f778882-34c7-4a55-af11-52686b6b5ea7" containerName="maas-api" Nov 03 18:41:23.206639 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.206617 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f778882-34c7-4a55-af11-52686b6b5ea7" containerName="maas-api" Nov 03 18:41:23.206845 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.206752 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f778882-34c7-4a55-af11-52686b6b5ea7" containerName="maas-api" Nov 03 18:41:23.210025 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.210003 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.285078 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.284911 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-8d4263-openshift-default-dockercfg-b86m7\"" Nov 03 18:41:23.317551 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.317518 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:41:23.348888 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.348848 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.348912 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349065 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349019 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349146 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349083 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349184 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349154 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8lg8\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349192 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349226 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349218 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349372 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349350 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.349439 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.349400 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450278 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450243 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450311 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450458 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450359 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8lg8\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450490 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450526 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450564 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450550 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450607 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450687 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450724 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.450769 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450728 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.451004 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.450964 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.451054 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.451019 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.451098 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.451067 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.451358 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.451335 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.453026 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.453003 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.453216 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.453197 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.514090 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.514031 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.514307 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.514282 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8lg8\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\") pod \"e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.558311 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.558270 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:23.997915 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:23.997888 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:41:23.999686 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:41:23.999657 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfeafaebe_fc71_40d7_a7ab_7de89b728833.slice/crio-4e2191d83e4ec4d3b540a86f4bb919a9a4462bbc8cc9c448d4acaa416016ad4e WatchSource:0}: Error finding container 4e2191d83e4ec4d3b540a86f4bb919a9a4462bbc8cc9c448d4acaa416016ad4e: Status 404 returned error can't find the container with id 4e2191d83e4ec4d3b540a86f4bb919a9a4462bbc8cc9c448d4acaa416016ad4e Nov 03 18:41:24.001808 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.001779 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:24.001872 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.001840 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:24.001909 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.001869 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:24.616340 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.616301 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerStarted","Data":"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d"} Nov 03 18:41:24.616770 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.616363 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerStarted","Data":"4e2191d83e4ec4d3b540a86f4bb919a9a4462bbc8cc9c448d4acaa416016ad4e"} Nov 03 18:41:24.741427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:24.741372 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podStartSLOduration=1.741353405 podStartE2EDuration="1.741353405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:41:24.739233732 +0000 UTC m=+4064.568317334" watchObservedRunningTime="2026-11-03 18:41:24.741353405 +0000 UTC m=+4064.570437007" Nov 03 18:41:25.559039 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:25.559004 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:25.564353 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:25.564318 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:25.621397 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:25.621365 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:25.622351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:25.622322 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:26.628455 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.628423 4980 generic.go:356] "Generic (PLEG): container finished" podID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerID="c30c6f54dd71ac24e9979539b7ddf3b7445cc015df4a9667d4c4403be9e938f3" exitCode=0 Nov 03 18:41:26.628954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.628490 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerDied","Data":"c30c6f54dd71ac24e9979539b7ddf3b7445cc015df4a9667d4c4403be9e938f3"} Nov 03 18:41:26.628954 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.628551 4980 scope.go:122] "RemoveContainer" containerID="f9b3b7fe4380b0f6f6533504501bd90898e6c914f9740c0a344d4dc56e895a97" Nov 03 18:41:26.727667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.727616 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:41:26.787176 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787150 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787208 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787278 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787351 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787312 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787365 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787438 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787515 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787484 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787519 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.787692 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.787551 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\") pod \"359cf870-4cef-45f4-97cb-ba93daa30bce\" (UID: \"359cf870-4cef-45f4-97cb-ba93daa30bce\") " Nov 03 18:41:26.788385 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.788342 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:41:26.788534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.788451 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:41:26.788534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.788498 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:41:26.788534 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.788525 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:41:26.788899 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.788870 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:41:26.790742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.790716 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:41:26.790742 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.790747 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "kube-api-access-49t5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:41:26.790926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.790771 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:41:26.790926 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.790811 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo" pod "359cf870-4cef-45f4-97cb-ba93daa30bce" (UID: "359cf870-4cef-45f4-97cb-ba93daa30bce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:41:26.889332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889300 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49t5j\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-kube-api-access-49t5j\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889332 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889329 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889340 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/359cf870-4cef-45f4-97cb-ba93daa30bce-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889349 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889357 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889367 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889377 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889384 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/359cf870-4cef-45f4-97cb-ba93daa30bce-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:26.889531 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:26.889392 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/359cf870-4cef-45f4-97cb-ba93daa30bce-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:41:27.634520 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:27.634488 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" Nov 03 18:41:27.634964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:27.634525 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r" event={"ID":"359cf870-4cef-45f4-97cb-ba93daa30bce","Type":"ContainerDied","Data":"e6a70acfdc54abe086684482c235428c3465c656e05a6f736e5113ec2b8e8c96"} Nov 03 18:41:27.634964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:27.634628 4980 scope.go:122] "RemoveContainer" containerID="c30c6f54dd71ac24e9979539b7ddf3b7445cc015df4a9667d4c4403be9e938f3" Nov 03 18:41:27.854990 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:27.854952 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:41:27.907675 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:27.907564 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-c188b2e5-openshift-default-59dc49bc7-jr29r"] Nov 03 18:41:28.294095 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:28.294059 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" path="/var/lib/kubelet/pods/359cf870-4cef-45f4-97cb-ba93daa30bce/volumes" Nov 03 18:41:31.200085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200051 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:41:31.200537 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200523 4980 cpu_manager.go:414] "containerMap: removing container" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.200613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200538 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.200613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200586 4980 cpu_manager.go:414] "containerMap: removing container" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.200613 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200594 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.200721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200697 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.200721 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.200712 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="359cf870-4cef-45f4-97cb-ba93daa30bce" containerName="istio-proxy" Nov 03 18:41:31.205369 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.205351 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.264905 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.264873 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:41:31.311122 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.311097 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-8d4263\"" Nov 03 18:41:31.355695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.340761 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd4vl\" (UniqueName: \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.355695 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.340901 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.441757 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.441721 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.441967 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.441803 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd4vl\" (UniqueName: \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.442482 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.442456 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.505009 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.504948 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd4vl\" (UniqueName: \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\") pod \"payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.518427 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.518406 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:31.897562 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:31.897488 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:41:31.900175 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:41:31.900145 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93ef3907_a952_4c47_933c_f69073caa2ae.slice/crio-5f5546de35982bdf348441d79258ecf86d5b58b74e0e0cdac048e9cf55804cd8 WatchSource:0}: Error finding container 5f5546de35982bdf348441d79258ecf86d5b58b74e0e0cdac048e9cf55804cd8: Status 404 returned error can't find the container with id 5f5546de35982bdf348441d79258ecf86d5b58b74e0e0cdac048e9cf55804cd8 Nov 03 18:41:32.662643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:32.662606 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" event={"ID":"93ef3907-a952-4c47-933c-f69073caa2ae","Type":"ContainerStarted","Data":"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf"} Nov 03 18:41:32.663106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:32.662667 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" event={"ID":"93ef3907-a952-4c47-933c-f69073caa2ae","Type":"ContainerStarted","Data":"5f5546de35982bdf348441d79258ecf86d5b58b74e0e0cdac048e9cf55804cd8"} Nov 03 18:41:32.663106 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:32.662834 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:32.783101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:32.783055 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" podStartSLOduration=1.783042124 podStartE2EDuration="1.783042124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-03 18:41:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:41:32.78077929 +0000 UTC m=+4072.609862904" watchObservedRunningTime="2026-11-03 18:41:32.783042124 +0000 UTC m=+4072.612125726" Nov 03 18:41:43.489966 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:43.489934 4980 scope.go:122] "RemoveContainer" containerID="f8ed310a2ee8bec8356654935884dcd3d1a2aaae5d2e44a20f3802da289919ff" Nov 03 18:41:43.498714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:43.498695 4980 scope.go:122] "RemoveContainer" containerID="98008ecce625fe507360b4a6fe0e7af918610c1b9f41079db1e5391918653a6d" Nov 03 18:41:43.681119 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:43.681079 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:41:44.519452 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:41:44.519416 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:41:50.059250 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:50.059212 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj"] Nov 03 18:41:50.109299 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:50.109264 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd97c760-76mxj"] Nov 03 18:41:50.293744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:50.293668 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e0e5351-c948-4a00-abaa-d868a519925e" path="/var/lib/kubelet/pods/1e0e5351-c948-4a00-abaa-d868a519925e/volumes" Nov 03 18:41:52.777037 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.776952 4980 generic.go:356] "Generic (PLEG): container finished" podID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerID="c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d" exitCode=0 Nov 03 18:41:52.777492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.777030 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerDied","Data":"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d"} Nov 03 18:41:52.777492 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.777433 4980 scope.go:122] "RemoveContainer" containerID="c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d" Nov 03 18:41:52.778739 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.778708 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:52.778869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.778796 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:52.778869 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:52.778846 4980 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"} Nov 03 18:41:53.789830 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:53.789782 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerStarted","Data":"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55"} Nov 03 18:41:54.562595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:54.560853 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:54.562595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:54.561033 4980 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 18:41:54.562595 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:54.561082 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 18:41:55.572093 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:55.572058 4980 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 18:41:55.572755 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:55.572722 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 18:41:55.621667 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:55.621637 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:56.569041 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:56.569000 4980 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 18:41:56.569337 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:56.569303 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 18:41:57.563050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:57.563016 4980 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 18:41:57.563678 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:57.563645 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 18:41:58.572027 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:58.571985 4980 patch_prober.go:29] interesting pod/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" start-of-body= Nov 03 18:41:58.572703 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:58.572668 4980 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.206:15021/healthz/ready\": dial tcp 10.132.0.206:15021: connect: connection refused" Nov 03 18:41:59.562652 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:59.562621 4980 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:41:59.563472 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:41:59.563456 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:42:14.520135 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:42:14.520094 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:42:44.520014 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:42:44.519979 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:43:14.521108 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:43:14.521074 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:43:24.512144 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:24.512052 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:43:29.513927 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.513861 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" podUID="93ef3907-a952-4c47-933c-f69073caa2ae" containerName="payload-pre-processing" containerID="cri-o://68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf" gracePeriod=25 Nov 03 18:43:29.772111 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.772053 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:43:29.843279 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.843250 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\") pod \"93ef3907-a952-4c47-933c-f69073caa2ae\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " Nov 03 18:43:29.843463 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.843320 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\" (UniqueName: \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\") pod \"93ef3907-a952-4c47-933c-f69073caa2ae\" (UID: \"93ef3907-a952-4c47-933c-f69073caa2ae\") " Nov 03 18:43:29.843640 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.843617 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume" pod "93ef3907-a952-4c47-933c-f69073caa2ae" (UID: "93ef3907-a952-4c47-933c-f69073caa2ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:43:29.845737 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.845716 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl" pod "93ef3907-a952-4c47-933c-f69073caa2ae" (UID: "93ef3907-a952-4c47-933c-f69073caa2ae"). InnerVolumeSpecName "kube-api-access-jd4vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:43:29.944182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.944145 4980 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93ef3907-a952-4c47-933c-f69073caa2ae-plugins-config-volume\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:29.944182 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:29.944176 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd4vl\" (UniqueName: \"kubernetes.io/projected/93ef3907-a952-4c47-933c-f69073caa2ae-kube-api-access-jd4vl\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:30.277901 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.277869 4980 generic.go:356] "Generic (PLEG): container finished" podID="93ef3907-a952-4c47-933c-f69073caa2ae" containerID="68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf" exitCode=0 Nov 03 18:43:30.278084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.277919 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" Nov 03 18:43:30.278084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.277938 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" event={"ID":"93ef3907-a952-4c47-933c-f69073caa2ae","Type":"ContainerDied","Data":"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf"} Nov 03 18:43:30.278084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.277978 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l" event={"ID":"93ef3907-a952-4c47-933c-f69073caa2ae","Type":"ContainerDied","Data":"5f5546de35982bdf348441d79258ecf86d5b58b74e0e0cdac048e9cf55804cd8"} Nov 03 18:43:30.278084 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.277999 4980 scope.go:122] "RemoveContainer" containerID="68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf" Nov 03 18:43:30.287761 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.287742 4980 scope.go:122] "RemoveContainer" containerID="68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf" Nov 03 18:43:30.288029 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:43:30.288011 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf\": container with ID starting with 68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf not found: ID does not exist" containerID="68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf" Nov 03 18:43:30.288101 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.288041 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf"} err="failed to get container status \"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf\": rpc error: code = NotFound desc = could not find container \"68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf\": container with ID starting with 68a7f6785b362df64e0c1ea0569e3dcceb5a478884da15395e330c3b6cd02bbf not found: ID does not exist" Nov 03 18:43:30.453157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.453123 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:43:30.458699 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:30.458670 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-8d4263-5d97875fbc-4db8l"] Nov 03 18:43:32.292204 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:32.292173 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93ef3907-a952-4c47-933c-f69073caa2ae" path="/var/lib/kubelet/pods/93ef3907-a952-4c47-933c-f69073caa2ae/volumes" Nov 03 18:43:41.279736 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:41.279654 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:43:41.285116 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:41.285094 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:43:41.336628 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:41.336606 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7cd56b8dbc-mpbwn_a82bcfea-add1-4518-b0cc-f9b6b9d61f7b/console-operator/2.log" Nov 03 18:43:41.341357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:41.341337 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-d8hsc_fe747a0c-5a1d-4db3-876a-f26e1852450e/ovn-acl-logging/0.log" Nov 03 18:43:43.535452 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:43.535413 4980 scope.go:122] "RemoveContainer" containerID="b4e63439282f8c2d49f5a9665a09b92dd1f8ad66979566c8b8cd56fba89d135d" Nov 03 18:43:43.544241 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:43.544221 4980 scope.go:122] "RemoveContainer" containerID="c5da74577a0b0c5451e30ab72bf3a23e3decf70f5829ff6ecf912a445301233d" Nov 03 18:43:44.519520 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:43:44.519486 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:43:45.303495 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:45.303463 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:43:45.304022 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:45.303696 4980 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" containerID="cri-o://2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55" gracePeriod=30 Nov 03 18:43:49.070544 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:49.070510 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6"] Nov 03 18:43:49.120368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:49.120338 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-h46t6"] Nov 03 18:43:50.292616 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.292558 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68e73e89-bff3-4cd8-8a80-f6093ad05b61" path="/var/lib/kubelet/pods/68e73e89-bff3-4cd8-8a80-f6093ad05b61/volumes" Nov 03 18:43:50.559357 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.559298 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:43:50.655012 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.654982 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655035 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655074 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655113 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655153 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655144 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655217 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655253 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655507 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655470 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:43:50.655566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655506 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.655566 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655538 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Nov 03 18:43:50.655720 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655549 4980 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\") pod \"feafaebe-fc71-40d7-a7ab-7de89b728833\" (UID: \"feafaebe-fc71-40d7-a7ab-7de89b728833\") " Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655774 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655819 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.655844 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.656028 4980 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-data\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.656041 4980 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-certs\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.656050 4980 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-workload-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.656060 4980 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/feafaebe-fc71-40d7-a7ab-7de89b728833-istiod-ca-cert\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.656085 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.656068 4980 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-credential-socket\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.658113 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.658086 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:43:50.658214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.658105 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Nov 03 18:43:50.658214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.658092 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Nov 03 18:43:50.658214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.658153 4980 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8" pod "feafaebe-fc71-40d7-a7ab-7de89b728833" (UID: "feafaebe-fc71-40d7-a7ab-7de89b728833"). InnerVolumeSpecName "kube-api-access-l8lg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Nov 03 18:43:50.757687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.757652 4980 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8lg8\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-kube-api-access-l8lg8\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.757687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.757685 4980 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-podinfo\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.757687 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.757696 4980 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-envoy\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:50.757913 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:50.757707 4980 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/feafaebe-fc71-40d7-a7ab-7de89b728833-istio-token\") on node \"ip-10-0-139-234.us-west-1.compute.internal\" DevicePath \"\"" Nov 03 18:43:51.407714 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.407679 4980 generic.go:356] "Generic (PLEG): container finished" podID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerID="2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55" exitCode=0 Nov 03 18:43:51.408208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.407731 4980 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" Nov 03 18:43:51.408208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.407771 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerDied","Data":"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55"} Nov 03 18:43:51.408208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.407843 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4" event={"ID":"feafaebe-fc71-40d7-a7ab-7de89b728833","Type":"ContainerDied","Data":"4e2191d83e4ec4d3b540a86f4bb919a9a4462bbc8cc9c448d4acaa416016ad4e"} Nov 03 18:43:51.408208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.407916 4980 scope.go:122] "RemoveContainer" containerID="2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55" Nov 03 18:43:51.418019 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.418003 4980 scope.go:122] "RemoveContainer" containerID="c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d" Nov 03 18:43:51.426313 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.426149 4980 scope.go:122] "RemoveContainer" containerID="2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55" Nov 03 18:43:51.426429 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:43:51.426412 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55\": container with ID starting with 2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55 not found: ID does not exist" containerID="2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55" Nov 03 18:43:51.426474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.426439 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55"} err="failed to get container status \"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55\": rpc error: code = NotFound desc = could not find container \"2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55\": container with ID starting with 2cfbaeeb8657efd79f8c2a5746ff975ab0f4084a173ed226237e7f10a600dd55 not found: ID does not exist" Nov 03 18:43:51.426474 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.426457 4980 scope.go:122] "RemoveContainer" containerID="c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d" Nov 03 18:43:51.426708 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:43:51.426692 4980 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d\": container with ID starting with c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d not found: ID does not exist" containerID="c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d" Nov 03 18:43:51.426761 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.426712 4980 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d"} err="failed to get container status \"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d\": rpc error: code = NotFound desc = could not find container \"c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d\": container with ID starting with c4f0dc8e62f94deeb79fa8c3a0eced126fb00653a11d7c83ae9bfa57f71f0e2d not found: ID does not exist" Nov 03 18:43:51.626164 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.626132 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:43:51.676391 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:51.676320 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-8d4263-openshift-default-5cfb874957-dkcg4"] Nov 03 18:43:52.293142 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:43:52.293110 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" path="/var/lib/kubelet/pods/feafaebe-fc71-40d7-a7ab-7de89b728833/volumes" Nov 03 18:44:14.519642 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:44:14.519604 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:44:26.057838 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:26.057803 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw"] Nov 03 18:44:26.108136 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:26.108101 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-45mgw"] Nov 03 18:44:26.323274 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:26.323194 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0b1e96c-c86b-456a-b3e5-3008409038ed" path="/var/lib/kubelet/pods/c0b1e96c-c86b-456a-b3e5-3008409038ed/volumes" Nov 03 18:44:44.518889 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:44:44.518848 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:44:59.062203 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:59.062120 4980 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx"] Nov 03 18:44:59.113253 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:59.113223 4980 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b923ecf3-gt9fx"] Nov 03 18:44:59.233786 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:44:59.233753 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbc5b6946-5pd45_d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d/authorino/0.log" Nov 03 18:45:00.295684 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:00.295644 4980 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4224210-7b9e-4add-82c0-7f586cc643cc" path="/var/lib/kubelet/pods/e4224210-7b9e-4add-82c0-7f586cc643cc/volumes" Nov 03 18:45:10.841030 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:10.840996 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ks4hn_f7f75137-d10f-4d81-8c4f-5b8612ee4354/manager/0.log" Nov 03 18:45:11.767075 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:11.767041 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:45:12.210144 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:12.210119 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g98gb_3c649050-308d-47b6-81ae-5bcf17531e05/server/0.log" Nov 03 18:45:12.448269 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:12.448231 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9m7h2_27aefc84-7981-428b-a86a-69863cac4ec3/manager/0.log" Nov 03 18:45:13.535381 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:13.535345 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7456bf56b5-ccl87_bb3bbb76-a54e-4aa8-bd73-a35af88de3b1/maas-api/0.log" Nov 03 18:45:14.522627 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:45:14.522592 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:45:14.953034 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:14.953003 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rk2t6_3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2/postgres/0.log" Nov 03 18:45:16.750634 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:16.750602 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbc5b6946-5pd45_d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d/authorino/0.log" Nov 03 18:45:19.793777 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:19.793746 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7svwg_e15395a8-9631-42fb-b9be-7eae057ebc92/discovery/0.log" Nov 03 18:45:20.387013 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:20.386980 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8f76b86f-nqp5r_5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d/kube-auth-proxy/0.log" Nov 03 18:45:20.634069 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:20.634032 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4cj7c_819d1443-6698-423f-911d-8cb12ea7a3c1/istio-proxy/0.log" Nov 03 18:45:21.434167 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:21.434138 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-bftjk_32c6c7af-e170-4045-bb87-4cca726a11f7/istio-proxy/0.log" Nov 03 18:45:22.463916 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:22.463883 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-r96r9_250bf139-b14a-4c76-8bf3-842f5be6494f/istio-proxy/0.log" Nov 03 18:45:22.788747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:22.788673 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-tcgv5_c53d3b05-4c4c-4c94-b156-684fd3e7435a/istio-proxy/0.log" Nov 03 18:45:24.041162 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:24.041134 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b97b666bc-r5q4p_6a6cf569-649e-420f-8fcf-3c1443782d9e/router/0.log" Nov 03 18:45:25.751820 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:25.751786 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f46zb_00efe73b-9c70-4536-9b6f-cbd8e4ca188d/main/0.log" Nov 03 18:45:26.026694 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:26.026614 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb_5172ab4a-c7f4-4fac-a164-b77afacc7c5c/main/0.log" Nov 03 18:45:26.304588 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:26.304501 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-j7n9l_a742aa19-68c7-4725-8b5c-0c988255eb17/main/0.log" Nov 03 18:45:26.525898 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:26.525869 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4_07ed2d59-e332-4426-bc90-87189e39ef2e/main/0.log" Nov 03 18:45:26.762252 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:26.762216 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj_bf5ad4c1-ae4c-487b-baa4-bf5385b1420e/main/0.log" Nov 03 18:45:27.703516 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:27.703482 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbc5b6946-5pd45_d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d/authorino/0.log" Nov 03 18:45:37.793553 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:37.793520 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-module-controller-manager-7566cf5fc8-ks4hn_f7f75137-d10f-4d81-8c4f-5b8612ee4354/manager/0.log" Nov 03 18:45:38.531490 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:38.531453 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:45:38.860375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:38.860286 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-g98gb_3c649050-308d-47b6-81ae-5bcf17531e05/server/0.log" Nov 03 18:45:39.101747 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:39.101717 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-9m7h2_27aefc84-7981-428b-a86a-69863cac4ec3/manager/0.log" Nov 03 18:45:40.036537 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:40.036501 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-7456bf56b5-ccl87_bb3bbb76-a54e-4aa8-bd73-a35af88de3b1/maas-api/0.log" Nov 03 18:45:41.333375 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:41.333297 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rk2t6_3f2200c5-a9a7-4dfa-a15e-1779c6fa02f2/postgres/0.log" Nov 03 18:45:42.578509 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:42.578477 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbc5b6946-5pd45_d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d/authorino/0.log" Nov 03 18:45:43.586296 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:43.586259 4980 scope.go:122] "RemoveContainer" containerID="257d7b4a3a5d7f79c32ed8d94092776cf90e41a8f70b3946836b7a3445a9f0f7" Nov 03 18:45:43.594980 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:43.594961 4980 scope.go:122] "RemoveContainer" containerID="8d21e33a484287166a443e93e81354dd9e6e1ec93099c17863e47c674740d667" Nov 03 18:45:44.519016 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:45:44.518985 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:45:45.216229 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:45.216198 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-7svwg_e15395a8-9631-42fb-b9be-7eae057ebc92/discovery/0.log" Nov 03 18:45:45.804131 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:45.804096 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7c8f76b86f-nqp5r_5c8ff94d-8f0f-49d2-ab9c-93c646bc6f9d/kube-auth-proxy/0.log" Nov 03 18:45:46.035148 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:46.035112 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-4cj7c_819d1443-6698-423f-911d-8cb12ea7a3c1/istio-proxy/0.log" Nov 03 18:45:46.818050 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:46.818012 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-bftjk_32c6c7af-e170-4045-bb87-4cca726a11f7/istio-proxy/0.log" Nov 03 18:45:47.850208 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:47.850175 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-r96r9_250bf139-b14a-4c76-8bf3-842f5be6494f/istio-proxy/0.log" Nov 03 18:45:48.138709 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:48.138631 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-5df48fcc99-tcgv5_c53d3b05-4c4c-4c94-b156-684fd3e7435a/istio-proxy/0.log" Nov 03 18:45:49.366528 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:49.366494 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-5b97b666bc-r5q4p_6a6cf569-649e-420f-8fcf-3c1443782d9e/router/0.log" Nov 03 18:45:50.733284 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:50.733253 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-f46zb_00efe73b-9c70-4536-9b6f-cbd8e4ca188d/main/0.log" Nov 03 18:45:51.007258 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:51.007184 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b8rrfnb_5172ab4a-c7f4-4fac-a164-b77afacc7c5c/main/0.log" Nov 03 18:45:51.288785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:51.288700 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-j7n9l_a742aa19-68c7-4725-8b5c-0c988255eb17/main/0.log" Nov 03 18:45:51.516835 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:51.516804 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-pqlz4_07ed2d59-e332-4426-bc90-87189e39ef2e/main/0.log" Nov 03 18:45:51.760083 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:45:51.760051 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5f5c4b459f-nmgnj_bf5ad4c1-ae4c-487b-baa4-bf5385b1420e/main/0.log" Nov 03 18:46:14.519001 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:46:14.518966 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:46:25.920033 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:46:25.919956 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:46:26.495056 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:46:26.495021 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:46:43.617368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:46:43.617329 4980 scope.go:122] "RemoveContainer" containerID="b14b667a354505cce6fdd666ea2731bd857ebd30c24154e3502deda543ec9e09" Nov 03 18:46:44.519885 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:46:44.519845 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:47:05.032532 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:05.032494 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:47:05.657276 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:05.657242 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-b7df46f9c-p85x2_fbfd47f6-aae9-4164-ae0b-6ed876a106ea/manager/0.log" Nov 03 18:47:14.522423 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:47:14.522370 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor" Nov 03 18:47:17.340426 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:17.340390 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-fnq58_fedfb5ca-94b5-4850-8020-a7279aa49eae/global-pull-secret-syncer/0.log" Nov 03 18:47:17.945368 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:17.945338 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-6tgbt_dfad8e5c-4000-47b2-b39c-ea805fd6e315/connection-checker/0.log" Nov 03 18:47:18.381425 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:18.381353 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-p4zh5_bea67a96-85b1-41d3-b177-bfec1d287457/connection-checker/0.log" Nov 03 18:47:18.811744 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:18.811714 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kas-connection-checker-685895c7c9-w5ndm_defb7da5-8103-40ef-9607-49c56c281187/connection-checker/0.log" Nov 03 18:47:20.155729 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:20.155690 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-vwkww_cad57e2f-34f9-4ad6-82d7-3ad939d0fef4/konnectivity-agent/0.log" Nov 03 18:47:21.493772 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:21.493741 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-234.us-west-1.compute.internal_32d0bc0f39e3274106efc0e449e45f7e/haproxy/0.log" Nov 03 18:47:26.243504 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.243468 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-cbc5b6946-5pd45_d5da6d0f-f0f5-49f2-a8ea-f4b2fa01291d/authorino/0.log" Nov 03 18:47:26.391785 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.391754 4980 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974"] Nov 03 18:47:26.392157 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392144 4980 cpu_manager.go:414] "containerMap: removing container" podUID="93ef3907-a952-4c47-933c-f69073caa2ae" containerName="payload-pre-processing" Nov 03 18:47:26.392214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392158 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93ef3907-a952-4c47-933c-f69073caa2ae" containerName="payload-pre-processing" Nov 03 18:47:26.392214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392173 4980 cpu_manager.go:414] "containerMap: removing container" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" Nov 03 18:47:26.392214 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392178 4980 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" Nov 03 18:47:26.392322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392250 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="93ef3907-a952-4c47-933c-f69073caa2ae" containerName="payload-pre-processing" Nov 03 18:47:26.392322 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.392257 4980 memory_manager.go:365] "RemoveStaleState removing state" podUID="feafaebe-fc71-40d7-a7ab-7de89b728833" containerName="istio-proxy" Nov 03 18:47:26.395366 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.395350 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.462936 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.462906 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-proc\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.463074 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.462958 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ngd6\" (UniqueName: \"kubernetes.io/projected/233e3ce9-f9bb-4f89-82ca-182184bbbd34-kube-api-access-6ngd6\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.463074 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.463048 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-podres\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.463155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.463100 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-lib-modules\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.463155 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.463143 4980 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-sys\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.488393 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.488366 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974"] Nov 03 18:47:26.538924 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.538843 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"openshift-service-ca.crt\"" Nov 03 18:47:26.539060 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.538938 4980 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vlnz7\"/\"kube-root-ca.crt\"" Nov 03 18:47:26.560964 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.560946 4980 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vlnz7\"/\"default-dockercfg-ngffq\"" Nov 03 18:47:26.564405 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564384 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ngd6\" (UniqueName: \"kubernetes.io/projected/233e3ce9-f9bb-4f89-82ca-182184bbbd34-kube-api-access-6ngd6\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564433 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-podres\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564488 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564451 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-lib-modules\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564537 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-lib-modules\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564545 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-podres\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564618 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564563 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-sys\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564625 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-sys\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564673 4980 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-proc\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.564734 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.564731 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/233e3ce9-f9bb-4f89-82ca-182184bbbd34-proc\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.627388 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.627356 4980 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ngd6\" (UniqueName: \"kubernetes.io/projected/233e3ce9-f9bb-4f89-82ca-182184bbbd34-kube-api-access-6ngd6\") pod \"perf-node-gather-daemonset-mn974\" (UID: \"233e3ce9-f9bb-4f89-82ca-182184bbbd34\") " pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:26.707623 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:26.707597 4980 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:27.086646 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:27.086418 4980 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974"] Nov 03 18:47:27.088919 ip-10-0-139-234 kubenswrapper[4980]: W1103 18:47:27.088893 4980 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod233e3ce9_f9bb_4f89_82ca_182184bbbd34.slice/crio-df55373f4fc1d7693e6baf3aab51817f88940f3ec76f69f4b8899c6cd5b97476 WatchSource:0}: Error finding container df55373f4fc1d7693e6baf3aab51817f88940f3ec76f69f4b8899c6cd5b97476: Status 404 returned error can't find the container with id df55373f4fc1d7693e6baf3aab51817f88940f3ec76f69f4b8899c6cd5b97476 Nov 03 18:47:27.577029 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:27.576985 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" event={"ID":"233e3ce9-f9bb-4f89-82ca-182184bbbd34","Type":"ContainerStarted","Data":"e6e6de1fec30872f4aab0f37183fe078d2ee288eaffaa0ec5278f121f48af4c2"} Nov 03 18:47:27.577461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:27.577051 4980 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" event={"ID":"233e3ce9-f9bb-4f89-82ca-182184bbbd34","Type":"ContainerStarted","Data":"df55373f4fc1d7693e6baf3aab51817f88940f3ec76f69f4b8899c6cd5b97476"} Nov 03 18:47:27.577461 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:27.577085 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:27.696451 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:27.696399 4980 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" podStartSLOduration=1.6963861709999999 podStartE2EDuration="1.696386171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-03 18:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-03 18:47:27.695082919 +0000 UTC m=+4427.524166520" watchObservedRunningTime="2026-11-03 18:47:27.696386171 +0000 UTC m=+4427.525469773" Nov 03 18:47:33.594264 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:33.594233 4980 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vlnz7/perf-node-gather-daemonset-mn974" Nov 03 18:47:34.455326 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:34.455282 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-884576c57-zzdc8_3ba272e4-ee5f-4abd-9c93-26f5a2563098/cluster-monitoring-operator/0.log" Nov 03 18:47:37.095643 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:37.095614 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vcdj_187d0aaf-1c85-42e0-be08-6bd41ffa9bab/node-exporter/0.log" Nov 03 18:47:37.515787 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:37.515760 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vcdj_187d0aaf-1c85-42e0-be08-6bd41ffa9bab/kube-rbac-proxy/0.log" Nov 03 18:47:37.936988 ip-10-0-139-234 kubenswrapper[4980]: I1103 18:47:37.936914 4980 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-8vcdj_187d0aaf-1c85-42e0-be08-6bd41ffa9bab/init-textfile/0.log" Nov 03 18:47:44.519856 ip-10-0-139-234 kubenswrapper[4980]: E1103 18:47:44.519815 4980 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-42zww" containerName="extractor"